[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519656015 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002227] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006302] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009167] LSM: Security Framework initializing [ 0.010038] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012053] *** VALIDATE selinux *** [ 0.020034] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024688] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025212] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027176] *** VALIDATE tmpfs *** [ 0.028835] *** VALIDATE proc *** [ 0.031071] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.034032] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037034] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041000] debug: unmapping init [mem 0xffffffffb7e59000-0xffffffffb7e60fff] [ 0.042209] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045240] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046033] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048007] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050008] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053380] rcu: Hierarchical SRCU implementation. [ 0.055630] smp: Bringing up secondary CPUs ... [ 0.056604] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.061013] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.280087] node 0 deferred pages initialised in 215ms [ 0.285223] devtmpfs: initialized [ 0.286774] x86/mm: Memory block size: 128MB [ 0.288799] gcov: version magic: 0x41383552 [ 0.290235] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.291081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.292355] pinctrl core: initialized pinctrl subsystem [ 0.293174] [ 0.293780] ************************************************************* [ 0.294010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.295010] ** ** [ 0.296008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.297009] ** ** [ 0.298010] ** This means that this kernel is built to expose internal ** [ 0.299007] ** IOMMU data structures, which may compromise security on ** [ 0.300008] ** your system. ** [ 0.301008] ** ** [ 0.302008] ** If you see this message and you are not debugging the ** [ 0.303008] ** kernel, report this immediately to your vendor! ** [ 0.304009] ** ** [ 0.305010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.306009] ************************************************************* [ 0.307539] NET: Registered protocol family 16 [ 0.308523] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.311044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.317029] cpuidle: using governor menu [ 0.319262] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.321602] PCI: Using configuration type 1 for base access [ 0.324133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.333733] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.335664] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340112] cryptd: max_cpu_qlen set to 1000 [ 0.346163] ACPI: Added _OSI(Module Device) [ 0.348009] ACPI: Added _OSI(Processor Device) [ 0.350010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.352011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.357682] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.365067] ACPI: Interpreter enabled [ 0.366882] ACPI: PM: (supports S0 S3 S4 S5) [ 0.368014] ACPI: Using IOAPIC for interrupt routing [ 0.370125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.373373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.383859] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.386035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.393163] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.398568] acpiphp: Slot [2] registered [ 0.400131] acpiphp: Slot [5] registered [ 0.402480] acpiphp: Slot [6] registered [ 0.403098] acpiphp: Slot [7] registered [ 0.405070] acpiphp: Slot [8] registered [ 0.406070] acpiphp: Slot [9] registered [ 0.408077] acpiphp: Slot [10] registered [ 0.409156] acpiphp: Slot [3] registered [ 0.412117] acpiphp: Slot [4] registered [ 0.413087] acpiphp: Slot [11] registered [ 0.415073] acpiphp: Slot [12] registered [ 0.416070] acpiphp: Slot [13] registered [ 0.417061] acpiphp: Slot [14] registered [ 0.419061] acpiphp: Slot [15] registered [ 0.421072] acpiphp: Slot [16] registered [ 0.422071] acpiphp: Slot [17] registered [ 0.424076] acpiphp: Slot [18] registered [ 0.425071] acpiphp: Slot [19] registered [ 0.426056] acpiphp: Slot [20] registered [ 0.427119] acpiphp: Slot [21] registered [ 0.429066] acpiphp: Slot [22] registered [ 0.430047] acpiphp: Slot [23] registered [ 0.431058] acpiphp: Slot [24] registered [ 0.432000] acpiphp: Slot [25] registered [ 0.432000] acpiphp: Slot [26] registered [ 0.434066] acpiphp: Slot [27] registered [ 0.435083] acpiphp: Slot [28] registered [ 0.436051] acpiphp: Slot [29] registered [ 0.437062] acpiphp: Slot [30] registered [ 0.438045] acpiphp: Slot [31] registered [ 0.438997] PCI host bridge to bus 0000:00 [ 0.440013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.445033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.448024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.451024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.454039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460102] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.465090] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.474762] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.480051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.482011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.484010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.486010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.489651] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.491730] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.494036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.497322] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.501000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.512014] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.517013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.522445] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.526019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.529018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.541018] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.550071] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.556016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.565017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.578016] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.586819] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614062] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631015] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.638000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.642011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.647018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.658019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.667906] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.682020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.701021] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.716323] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.736021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.768033] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.781735] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.784324] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.786337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.788305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.790196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.796788] iommu: Default domain type: Passthrough [ 0.797865] SCSI subsystem initialized [ 0.798099] ACPI: bus type USB registered [ 0.799105] usbcore: registered new interface driver usbfs [ 0.800058] usbcore: registered new interface driver hub [ 0.802082] usbcore: registered new device driver usb [ 0.803142] pps_core: LinuxPPS API ver. 1 registered [ 0.804000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.804070] PTP clock support registered [ 0.805085] EDAC MC: Ver: 3.0.0 [ 0.807388] PCI: Using ACPI for IRQ routing [ 0.812352] NetLabel: Initializing [ 0.813009] NetLabel: domain hash size = 128 [ 0.819013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.823217] NetLabel: unlabeled traffic allowed by default [ 0.824316] vgaarb: loaded [ 0.826432] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.827010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.836497] clocksource: Switched to clocksource kvm-clock [ 1.005933] VFS: Disk quotas dquot_6.6.0 [ 1.007666] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.010270] *** VALIDATE ramfs *** [ 1.011506] *** VALIDATE hugetlbfs *** [ 1.012988] pnp: PnP ACPI init [ 1.015492] pnp: PnP ACPI: found 6 devices [ 1.033104] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.036220] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.038420] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.040091] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.041883] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.043594] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.045754] NET: Registered protocol family 2 [ 1.047769] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.051545] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.054497] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.065225] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.070051] TCP: Hash tables configured (established 65536 bind 65536) [ 1.073247] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.076197] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078749] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.082027] NET: Registered protocol family 1 [ 1.085823] RPC: Registered named UNIX socket transport module. [ 1.090445] RPC: Registered udp transport module. [ 1.092033] RPC: Registered tcp transport module. [ 1.094555] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.096375] NET: Registered protocol family 44 [ 1.097908] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.099982] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.102048] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.104639] PCI: CLS 0 bytes, default 64 [ 1.106380] Unpacking initramfs... [ 2.699750] debug: unmapping init [mem 0xffff9bd13cc54000-0xffff9bd13ffbffff] [ 2.703792] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.706107] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.710385] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.208393] Initialise system trusted keyrings [ 3.209803] Key type blacklist registered [ 3.213224] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.221694] zbud: loaded [ 3.226077] *** VALIDATE nfs *** [ 3.227121] *** VALIDATE nfs4 *** [ 3.228378] pstore: using deflate compression [ 3.232339] Platform Keyring initialized [ 3.372982] NET: Registered protocol family 38 [ 3.374935] Key type asymmetric registered [ 3.376179] Asymmetric key parser 'x509' registered [ 3.378146] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.381694] io scheduler mq-deadline registered [ 3.383791] io scheduler kyber registered [ 3.385741] io scheduler bfq registered [ 3.387704] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.391688] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.394120] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.396676] ACPI: Power Button [PWRF] [ 3.499692] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.913851] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.328018] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.446101] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.660208] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.695649] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.737928] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.743278] Non-volatile memory driver v1.3 [ 4.744795] Linux agpgart interface v0.103 [ 4.774847] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 4.778104] vda: detected capacity change from 0 to 67940352 [ 4.800486] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.803619] vdb: detected capacity change from 0 to 1073741824 [ 4.825678] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.830232] vdc: detected capacity change from 0 to 2621440000 [ 4.851347] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.854368] vdd: detected capacity change from 0 to 2621440000 [ 4.871713] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.874511] vde: detected capacity change from 0 to 4294967296 [ 4.889286] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.892494] vdf: detected capacity change from 0 to 4294967296 [ 4.898424] libphy: Fixed MDIO Bus: probed [ 4.904719] usbcore: registered new interface driver usbserial_generic [ 4.906989] usbserial: USB Serial support registered for generic [ 4.909190] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.913178] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.914927] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.917509] mousedev: PS/2 mouse device common for all mice [ 4.922190] rtc_cmos 00:05: RTC can wake from S4 [ 4.925712] rtc_cmos 00:05: registered as rtc0 [ 4.927262] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.932494] intel_pstate: CPU model not supported [ 4.937312] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.938649] hid: raw HID events driver (C) Jiri Kosina [ 4.944637] usbcore: registered new interface driver usbhid [ 4.950123] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.953717] usbhid: USB HID core driver [ 4.953887] drop_monitor: Initializing network drop monitor service [ 4.954032] Initializing XFRM netlink socket [ 4.954434] NET: Registered protocol family 10 [ 4.955599] Segment Routing with IPv6 [ 4.955630] NET: Registered protocol family 17 [ 4.956536] mpls_gso: MPLS GSO support [ 4.975315] RAS: Correctable Errors collector initialized. [ 4.978420] AVX version of gcm_enc/dec engaged. [ 4.980481] AES CTR mode by8 optimization enabled [ 4.983886] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.083139] sched_clock: Marking stable (5083118534, 0)->(6127687558, -1044569024) [ 5.090537] registered taskstats version 1 [ 5.093227] Loading compiled-in X.509 certificates [ 5.095029] zswap: loaded using pool lzo/zbud [ 5.136140] Key type big_key registered [ 5.155467] Key type encrypted registered [ 5.157659] ima: No TPM chip found, activating TPM-bypass! [ 5.160207] ima: Allocated hash algorithm: sha1 [ 5.161672] ima: No architecture policies found [ 5.163721] evm: Initialising EVM extended attributes: [ 5.165470] evm: security.selinux [ 5.166727] evm: security.ima [ 5.167780] evm: security.capability [ 5.169345] evm: HMAC attrs: 0x1 [ 5.172228] rtc_cmos 00:05: setting system clock to 2025-07-24 13:40:33 UTC (1753364433) [ 5.179422] debug: unmapping init [mem 0xffffffffb8e03000-0xffffffffb8ffffff] [ 5.182537] debug: unmapping init [mem 0xffffffffb7b82000-0xffffffffb7e58fff] [ 5.192116] Write protecting the kernel read-only data: 28672k [ 5.195050] debug: unmapping init [mem 0xffffffffb6203000-0xffffffffb63fffff] [ 5.197468] debug: unmapping init [mem 0xffffffffb6b14000-0xffffffffb6bfffff] [ 5.245650] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.255412] systemd[1]: Detected virtualization kvm. [ 5.260989] systemd[1]: Detected architecture x86-64. [ 5.262920] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.290195] systemd[1]: No hostname configured. [ 5.292102] systemd[1]: Set hostname to . [ 5.294106] random: systemd: uninitialized urandom read (16 bytes read) [ 5.296269] systemd[1]: Initializing machine ID from random generator. [ 5.494051] random: ln: uninitialized urandom read (6 bytes read) [ 5.859268] random: systemd: uninitialized urandom read (16 bytes read) [ 5.864932] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.871560] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.881470] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.355067] device-mapper: uevent: version 1.0.3 [ 7.359500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.864100] random: fast init done [ 8.870261] virtio_net virtio0 ens2: renamed from eth0 [ 9.334435] scsi host0: ata_piix [ 9.379434] scsi host1: ata_piix [ 9.380770] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.391454] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.687756] random: crng init done [ 13.689070] random: 7 urandom warning(s) missed due to ratelimiting [ 14.728839] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.421571] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.866526] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.468731] SELinux: Disabled at runtime. [ 17.555190] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.566401] systemd[1]: Detected virtualization kvm. [ 17.569295] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.698952] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.706884] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.718410] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.724655] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.736193] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.747496] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.760761] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ 19.163297] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.651796] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.323806] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.355853] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.454495] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.477151] EDAC sbridge: Ver: 1.1.2 [ 21.734070] hrtimer: interrupt took 2620389 ns [ 24.012303] Key type dns_resolver registered [ 24.646187] NFS: Registering the id_resolver key type [ 24.648734] Key type id_resolver registered [ 24.650432] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg123-server login: [ 49.366458] libcfs: loading out-of-tree module taints kernel. [ 49.375689] Key type ._llcrypt registered [ 49.376561] Key type .llcrypt registered [ 49.409496] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 56.649387] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 57.198521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.206826] alg: No test for adler32 (adler32-zlib) [ 58.217984] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 58.579454] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 58.581652] LNet: Accept secure, port 988 [ 60.207150] Key type lgssc registered [ 60.682066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.571097] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 68.836685] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.718447] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.063942] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.526334] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.176699] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 86.613448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.646695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.672310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.825315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.847315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.900830] Lustre: lustre-MDT0000: new disk, initializing [ 87.968680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.982565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 90.292444] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.387210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.448916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.488133] Lustre: 6451:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.508332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.511477] Lustre: Skipped 1 previous similar message [ 99.570529] Lustre: lustre-MDT0001: new disk, initializing [ 99.638080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.656163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.662838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 102.095928] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.422468] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 111.222492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.285451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.505901] Lustre: lustre-OST0000: new disk, initializing [ 111.508863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 111.558408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.565480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 113.571534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 113.607503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 114.981573] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.887767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.946678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 124.011209] Lustre: lustre-OST0001: new disk, initializing [ 124.014868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 124.069921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 128.888593] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.560799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 129.565834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 129.657769] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 137.248179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 143.608204] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 150.966522] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing check_logdir /tmp/testlogs/ [ 153.121886] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing yml_node [ 155.118654] Lustre: DEBUG MARKER: Client: 2.16.57.3 [ 156.106412] Lustre: DEBUG MARKER: MDS: 2.16.57.3 [ 157.128290] Lustre: DEBUG MARKER: OSS: 2.16.57.3 [ 157.747560] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 24 09:43:06 EDT 2025 [ 165.350145] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 166.109577] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 180.704626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 180.704980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 180.710952] Lustre: Skipped 3 previous similar messages [ 180.714012] Lustre: Skipped 2 previous similar messages [ 184.616659] LustreError: 12115:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 184.681543] Lustre: server umount lustre-MDT0000 complete [ 186.544881] LustreError: 6442:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364614 with bad export cookie 6109949899098866651 [ 186.545497] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 186.562377] LustreError: 6442:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 190.944740] LustreError: 7738:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 190.944826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.946086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.956727] LustreError: 7738:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 190.972217] Lustre: Skipped 1 previous similar message [ 191.968380] LustreError: 6459:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 191.970230] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 191.980801] LustreError: 6459:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 191.984906] Lustre: Skipped 2 previous similar messages [ 192.721416] LustreError: 12318:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 192.724593] LustreError: 12318:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 192.843330] Lustre: server umount lustre-MDT0001 complete [ 195.114963] LustreError: 12520:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 195.118915] LustreError: 12520:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 195.158992] Lustre: server umount lustre-OST0000 complete [ 197.122624] LustreError: 12720:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 197.126266] LustreError: 12720:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 197.199327] Lustre: server umount lustre-OST0001 complete [ 200.520861] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 205.362994] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 210.798559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 214.662329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 217.798485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.005443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.630681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.656955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.783748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.804352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.850379] Lustre: lustre-MDT0000: new disk, initializing [ 224.905821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.918602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.773638] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 231.485367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.517665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 231.566141] Lustre: 14885:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.698064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.514645] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 237.884033] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 239.747851] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 243.389417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 243.433096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.571836] Lustre: lustre-OST0000: new disk, initializing [ 243.573751] Lustre: Skipped 1 previous similar message [ 243.576404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.579445] Lustre: Skipped 2 previous similar messages [ 243.625706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.628976] Lustre: Skipped 1 previous similar message [ 245.579770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.583817] Lustre: Skipped 1 previous similar message [ 245.586508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.661985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.482745] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 250.163311] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 251.910372] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 252.037962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 253.750216] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 253.837305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 255.969234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 255.970169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.972770] LustreError: Skipped 1 previous similar message [ 255.977994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 255.981222] Lustre: Skipped 1 previous similar message [ 260.716223] LustreError: 17793:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 260.718391] LustreError: 17793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 260.788047] Lustre: server umount lustre-OST0000 complete [ 262.624949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.627659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.632991] Lustre: Skipped 1 previous similar message [ 262.636056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.351497] Lustre: server umount lustre-MDT0000 complete [ 270.115638] LustreError: 15661:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364698 with bad export cookie 6109949899098868177 [ 270.119577] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.125266] LustreError: 15661:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 270.211231] LustreError: 18198:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 270.213286] LustreError: 18198:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 270.322810] Lustre: server umount lustre-MDT0001 complete [ 272.669521] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:45:00 (1753364700) [ 280.967503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 281.218873] LustreError: 19332:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.259668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 283.144062] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 286.690380] LustreError: 19332:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.933691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 290.595209] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 293.755767] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 295.774858] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 296.771980] Lustre: Failing over lustre-MDT0000 [ 296.853879] LustreError: 20662:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 296.857273] LustreError: 20662:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 296.884681] Lustre: server umount lustre-MDT0000 complete [ 297.825707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.829606] LustreError: 19326:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.832920] Lustre: Skipped 2 previous similar messages [ 297.842056] LustreError: 19326:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 308.192354] LustreError: 19332:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.198924] LustreError: 19332:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 311.946306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.016203] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.197725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 313.200113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 313.224651] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 314.285424] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 318.557675] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 320.986607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.987698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.993744] Lustre: Skipped 1 previous similar message [ 320.998824] Lustre: Skipped 2 previous similar messages [ 335.327127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.378478] LustreError: 21634:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 335.382196] LustreError: 21634:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 335.411552] Lustre: server umount lustre-MDT0000 complete [ 336.886511] LustreError: 20663:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364765 with bad export cookie 6109949899098869598 [ 336.889461] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.891988] LustreError: 20663:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 337.054216] Lustre: server umount lustre-MDT0001 complete [ 340.587942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.853706] LustreError: 22315:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.894967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.897800] Lustre: Skipped 2 previous similar messages [ 342.536862] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 346.223743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 348.102679] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 350.345223] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 351.870732] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 352.911491] Lustre: Failing over lustre-MDT0000 [ 353.016188] Lustre: server umount lustre-MDT0000 complete [ 356.000755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.152232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.232798] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.060312] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 371.426088] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 371.430620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 371.436136] Lustre: Skipped 1 previous similar message [ 371.444743] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 373.167937] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 376.801554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 376.803187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.810142] Lustre: Skipped 7 previous similar messages [ 381.795347] Lustre: server umount lustre-MDT0000 complete [ 381.921594] LustreError: 22310:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.928295] LustreError: 22310:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [ 383.286297] LustreError: 23028:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364811 with bad export cookie 6109949899098870529 [ 383.290681] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.291150] LustreError: 23028:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 383.450443] Lustre: server umount lustre-MDT0001 complete [ 386.951240] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:46:55 (1753364815) [ 394.231319] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 399.082890] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 404.529761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 407.557946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 410.326739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 413.211419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 417.915442] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 423.207420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 423.242420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.361410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 423.379318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 423.419631] Lustre: lustre-MDT0000: new disk, initializing [ 423.469061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.472130] Lustre: Skipped 2 previous similar messages [ 423.484757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 425.483243] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 431.887285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 431.934295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 432.001551] Lustre: 28256:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.193533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.413338] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 438.760370] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 440.546194] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 444.129250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 444.166664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 446.039111] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 446.066962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 446.074936] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 447.214973] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 451.516275] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 456.034105] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 466.402062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.408311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.422282] Lustre: Skipped 3 previous similar messages [ 466.426988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.410092] LustreError: 30873:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 467.412368] LustreError: 30873:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 467.500808] Lustre: server umount lustre-OST0000 complete [ 475.455466] Lustre: server umount lustre-MDT0000 complete [ 477.155255] LustreError: 28267:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.168297] LustreError: 28267:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 477.327343] LustreError: 28248:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364905 with bad export cookie 6109949899098870928 [ 477.330244] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.334466] LustreError: 28248:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.630213] Lustre: server umount lustre-MDT0001 complete [ 483.232140] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 484.503702] Key type lgssc unregistered [ 484.656412] LNet: 31801:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.669528] LNet: Removed LNI 192.168.201.123@tcp [ 486.087265] Key type .llcrypt unregistered [ 486.089086] Key type ._llcrypt unregistered [ 496.617935] Key type ._llcrypt registered [ 496.619321] Key type .llcrypt registered [ 496.671562] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 497.045971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 497.083088] alg: No test for adler32 (adler32-zlib) [ 497.967458] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 498.068771] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 498.071285] LNet: Accept secure, port 988 [ 499.663160] Key type lgssc registered [ 500.025827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 502.881801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 502.889888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.072933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.561192] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 508.539589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 508.692069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 510.082802] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 512.245456] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 513.660706] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 518.754571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 519.055241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.019458] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 527.465540] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 533.432099] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 545.206420] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:49:32 (1753364972) [ 546.821340] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 548.184631] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 553.444929] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:49:41 (1753364981) [ 560.711687] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:49:48 (1753364988) [ 561.405464] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 567.087193] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 567.091051] Lustre: Skipped 2 previous similar messages [ 568.057147] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:49:56 (1753364996) [ 572.049318] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 572.053190] Lustre: Skipped 4 previous similar messages [ 579.371358] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:50:07 (1753365007) [ 580.705973] Lustre: *** cfs_fail_loc=131b, val=2*** [ 582.723828] Lustre: *** cfs_fail_loc=131b, val=2*** [ 586.837843] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:50:15 (1753365015) [ 587.675711] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 587.678082] Lustre: Skipped 9 previous similar messages [ 593.609642] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:50:21 (1753365021) [ 600.034877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.036673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 600.048068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.059243] Lustre: Skipped 1 previous similar message [ 604.527406] LustreError: 39694:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 604.609677] Lustre: server umount lustre-OST0000 complete [ 610.272761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.273592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.278200] Lustre: Skipped 2 previous similar messages [ 612.548591] LustreError: 39897:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 612.551224] LustreError: 39897:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 612.596404] Lustre: server umount lustre-MDT0000 complete [ 614.645714] LustreError: 37223:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365042 with bad export cookie 1749410083866783768 [ 614.649277] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.651443] LustreError: 37223:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 614.744152] LustreError: 40098:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 614.751663] LustreError: 40098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 614.865620] Lustre: server umount lustre-MDT0001 complete [ 621.615855] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 623.547892] Key type lgssc unregistered [ 623.727312] LNet: 40623:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.806876] LNet: Removed LNI 192.168.201.123@tcp [ 625.299750] Key type .llcrypt unregistered [ 625.301058] Key type ._llcrypt unregistered [ 635.476953] Key type ._llcrypt registered [ 635.478830] Key type .llcrypt registered [ 635.537385] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 643.966404] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 644.493256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 644.523361] alg: No test for adler32 (adler32-zlib) [ 645.493740] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 645.623827] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 645.627620] LNet: Accept secure, port 988 [ 647.247671] Key type lgssc registered [ 647.868165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.039211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 656.640787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 660.100064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 664.191255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 667.975043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 668.023360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.037501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.166145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 669.195175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 669.263233] Lustre: lustre-MDT0000: new disk, initializing [ 669.322779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.341159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 671.776480] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 678.314755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 678.364562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 678.424973] Lustre: 43711:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 678.478263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 678.481690] Lustre: Skipped 1 previous similar message [ 678.530286] Lustre: lustre-MDT0001: new disk, initializing [ 678.599713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 678.634223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 678.639404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 680.757566] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 685.357305] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 687.341017] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 691.138562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 691.192179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 691.348876] Lustre: lustre-OST0000: new disk, initializing [ 691.352219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 691.419964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.081110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 692.086660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 692.141519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 694.867537] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 699.579906] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 701.925027] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 702.070862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 704.421196] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 704.554874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 707.557612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 707.558517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.560607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 707.561737] LustreError: Skipped 1 previous similar message [ 707.570318] Lustre: Skipped 1 previous similar message [ 712.672959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 712.683965] Lustre: Skipped 2 previous similar messages [ 717.796142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 717.799082] Lustre: Skipped 1 previous similar message [ 719.844374] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 719.868388] LustreError: 46619:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 719.960918] Lustre: server umount lustre-OST0000 complete [ 722.915139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.917322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.919370] Lustre: Skipped 1 previous similar message [ 728.033421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.037445] Lustre: Skipped 2 previous similar messages [ 736.223131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 736.273635] LustreError: 46821:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 736.276746] LustreError: 46821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 736.318656] Lustre: server umount lustre-MDT0000 complete [ 738.008227] LustreError: 43702:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365166 with bad export cookie 2782736926577828702 [ 738.010856] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 738.013038] LustreError: 43702:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 738.102461] LustreError: 47023:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 738.108398] LustreError: 47023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 738.252224] Lustre: server umount lustre-MDT0001 complete [ 742.543791] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:52:50 (1753365170) [ 745.371180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.588853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.335542] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 751.073499] LustreError: 47742:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 751.082077] LustreError: 47742:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 751.789759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 751.951241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 753.990437] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 756.730583] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 758.691334] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 762.418069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 762.595702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.877719] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 768.696451] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 781.919651] LustreError: 49938:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 782.367747] Lustre: Modifying parameter general.timeout=1 in log params [ 783.520987] Lustre: Modifying parameter general.timeout=4 in log params [ 783.524122] Lustre: Skipped 2 previous similar messages [ 785.682271] Lustre: Modifying parameter general.timeout=9 in log params [ 785.684393] Lustre: Skipped 4 previous similar messages [ 789.770226] Lustre: Modifying parameter general.timeout=20 in log params [ 789.773406] Lustre: Skipped 10 previous similar messages [ 798.115224] Lustre: Modifying parameter general.timeout=43 in log params [ 798.117568] Lustre: Skipped 22 previous similar messages [ 814.179464] Lustre: Modifying parameter general.timeout=86 in log params [ 814.182355] Lustre: Skipped 42 previous similar messages [ 834.334832] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:54:22 (1753365262) [ 835.491268] Lustre: 56841:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 836.948964] Lustre: 57037:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 836.953732] Lustre: 57037:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 839.046522] Lustre: 57353:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 839.053174] Lustre: 57353:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 2 previous similar messages [ 846.216401] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 846.220975] Lustre: Skipped 62 previous similar messages [ 849.581259] Lustre: 58685:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 849.589547] LustreError: 58685:0:(mgs_handler.c:990:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 851.139584] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:54:39 (1753365279) [ 859.616477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 859.619756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.625371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.628529] Lustre: Skipped 2 previous similar messages [ 861.249090] LustreError: 59379:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 861.251993] LustreError: 59379:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 861.286550] Lustre: server umount lustre-MDT0000 complete [ 862.661334] LustreError: 47766:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365290 with bad export cookie 2782736926577829584 [ 862.664780] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 862.667030] LustreError: 47766:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 862.823104] Lustre: server umount lustre-MDT0001 complete [ 874.484981] LustreError: 59782:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 874.487623] LustreError: 59782:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 874.566552] Lustre: server umount lustre-OST0000 complete [ 878.174483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 879.798434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 881.450756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 883.269468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 885.843300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.978610] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 886.096715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.618434] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 891.194094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 891.265747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 891.283349] Lustre: 61230:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 893.016559] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 895.330451] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.899796] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 899.707857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 899.776882] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 902.118112] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 904.332775] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 910.192593] Lustre: 63446:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 910.894147] Lustre: Modifying parameter general.jobid_var=disable in log params [ 910.896762] Lustre: Skipped 6 previous similar messages [ 915.429033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.429557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.432443] LustreError: Skipped 1 previous similar message [ 915.437314] Lustre: Skipped 2 previous similar messages [ 915.439799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.443585] Lustre: Skipped 4 previous similar messages [ 918.445224] LustreError: 63693:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 918.447373] LustreError: 63693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 918.533889] Lustre: server umount lustre-OST0000 complete [ 920.544498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.549797] Lustre: Skipped 1 previous similar message [ 926.301354] Lustre: server umount lustre-MDT0000 complete [ 927.782390] LustreError: 61950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365356 with bad export cookie 2782736926577831922 [ 927.785264] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.787524] LustreError: 61950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 927.950226] Lustre: server umount lustre-MDT0001 complete [ 933.179705] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 934.466267] Key type lgssc unregistered [ 934.619518] LNet: 64628:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.655734] LNet: Removed LNI 192.168.201.123@tcp [ 936.013485] Key type .llcrypt unregistered [ 936.015097] Key type ._llcrypt unregistered [ 946.128230] Key type ._llcrypt registered [ 946.129642] Key type .llcrypt registered [ 946.176063] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 946.541853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.602813] alg: No test for adler32 (adler32-zlib) [ 947.493650] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 947.592641] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 947.595753] LNet: Accept secure, port 988 [ 949.191187] Key type lgssc registered [ 949.654889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.948041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.958510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.173153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.627838] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 959.064637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 959.337642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 960.988544] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 963.297456] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 964.751695] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 967.530437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 967.758989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.849193] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 971.987164] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 985.439361] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 989.665685] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.666964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.671587] Lustre: Skipped 2 previous similar messages [ 989.673937] Lustre: Skipped 1 previous similar message [ 994.174255] LustreError: 68699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 994.202730] Lustre: server umount lustre-MDT0000 complete [ 994.784158] LustreError: 68137:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.790896] LustreError: 68137:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 995.553821] LustreError: 65968:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365423 with bad export cookie 5878254285178557672 [ 995.555918] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.558281] LustreError: 65968:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 995.634726] LustreError: 68902:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 995.637320] LustreError: 68902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 995.722069] Lustre: server umount lustre-MDT0001 complete [ 1007.409648] LustreError: 69103:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1007.412174] LustreError: 69103:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1007.480874] Lustre: server umount lustre-OST0000 complete [ 1010.499046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.856234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1013.312380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1014.787911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.387933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.491282] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1017.503893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1017.604120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.991132] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1022.177185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1022.216392] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1022.230661] Lustre: 70552:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1022.401098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.847178] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1025.716151] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.922340] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.141169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1029.203582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1031.247131] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1032.952850] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1033.445683] Lustre: 72775:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1034.065959] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1037.794070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.794894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.799795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.802063] LustreError: Skipped 1 previous similar message [ 1037.807159] Lustre: Skipped 2 previous similar messages [ 1041.387928] LustreError: 73026:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1041.390021] LustreError: 73026:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1041.450345] Lustre: server umount lustre-OST0000 complete [ 1042.912281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1042.912442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.912973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.919552] Lustre: Skipped 2 previous similar messages [ 1048.031897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.034644] Lustre: Skipped 2 previous similar messages [ 1048.769701] LustreError: 73233:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1048.771923] LustreError: 73233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1048.802847] Lustre: server umount lustre-MDT0000 complete [ 1050.084046] LustreError: 70544:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365478 with bad export cookie 5878254285178558491 [ 1050.086860] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.087803] LustreError: 70544:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1050.232670] Lustre: server umount lustre-MDT0001 complete [ 1054.875900] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1055.995258] Key type lgssc unregistered [ 1056.137397] LNet: 73960:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.189432] LNet: Removed LNI 192.168.201.123@tcp [ 1057.529749] Key type .llcrypt unregistered [ 1057.531382] Key type ._llcrypt unregistered [ 1067.119622] Key type ._llcrypt registered [ 1067.121138] Key type .llcrypt registered [ 1067.177114] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1067.570435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.591525] alg: No test for adler32 (adler32-zlib) [ 1068.462959] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 1068.561161] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1068.564549] LNet: Accept secure, port 988 [ 1070.159155] Key type lgssc registered [ 1070.606602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1073.794070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1073.803231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.036928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.668657] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1080.006027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1080.269607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.925905] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1084.408835] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1085.983092] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1089.249688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1089.529915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.700880] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1093.533884] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1106.114324] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1106.802727] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1106.805339] Lustre: Skipped 1 previous similar message [ 1107.873664] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1107.876582] Lustre: Skipped 2 previous similar messages [ 1109.937199] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1109.939877] Lustre: Skipped 5 previous similar messages [ 1114.003633] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1114.006801] Lustre: Skipped 10 previous similar messages [ 1122.217087] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1122.220015] Lustre: Skipped 23 previous similar messages [ 1138.445313] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1138.448208] Lustre: Skipped 45 previous similar messages [ 1143.985645] Lustre: 83053:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1144.915678] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:59:33 (1753365573) [ 1145.484766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1146.155811] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:59:34 (1753365574) [ 1150.945598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.946678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.955094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.957439] Lustre: Skipped 1 previous similar message [ 1153.389444] LustreError: 83345:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1153.456875] Lustre: server umount lustre-OST0000 complete [ 1156.065397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.066122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.071618] Lustre: Skipped 2 previous similar messages [ 1161.184457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.187092] Lustre: Skipped 2 previous similar messages [ 1166.305039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.307548] Lustre: Skipped 1 previous similar message [ 1169.375158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1169.412356] LustreError: 83549:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1169.414671] LustreError: 83549:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1169.443345] Lustre: server umount lustre-MDT0000 complete [ 1170.828528] LustreError: 78758:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365599 with bad export cookie 4731966774923478946 [ 1170.833367] LustreError: 78758:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1170.833908] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.901761] LustreError: 83753:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1170.904370] LustreError: 83753:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1171.008465] Lustre: server umount lustre-MDT0001 complete [ 1176.056367] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1177.323969] Key type lgssc unregistered [ 1177.469559] LNet: 84279:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.533404] LNet: Removed LNI 192.168.201.123@tcp [ 1178.910232] Key type .llcrypt unregistered [ 1178.912085] Key type ._llcrypt unregistered [ 1180.626079] Key type ._llcrypt registered [ 1180.627698] Key type .llcrypt registered [ 1180.683307] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1182.254674] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules [ 1182.715670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.773061] alg: No test for adler32 (adler32-zlib) [ 1183.653115] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 1183.744877] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1183.747371] LNet: Accept secure, port 988 [ 1185.343526] Key type lgssc registered [ 1185.825505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.317808] LustreError: 85379:0:(lu_object.c:1385:lu_context_key_register()) cfs_race id 60d sleeping [ 1194.381972] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1197.980745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.486930] LustreError: 85379:0:(lu_object.c:1385:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1207.499619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.744552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.159713] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1213.072527] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:00:41 (1753365641) [ 1214.726234] LustreError: 86410:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1214.730795] LustreError: 86410:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1214.733934] LustreError: 86410:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1214.799316] LustreError: 87176:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1214.893357] Lustre: server umount lustre-MDT0000 complete [ 1219.601428] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1220.704829] Key type lgssc unregistered [ 1220.850438] LNet: 87857:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.861419] LNet: Removed LNI 192.168.201.123@tcp [ 1222.167624] Key type .llcrypt unregistered [ 1222.168736] Key type ._llcrypt unregistered [ 1229.309542] Key type ._llcrypt registered [ 1229.311016] Key type .llcrypt registered [ 1229.357908] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1229.761581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.809862] alg: No test for adler32 (adler32-zlib) [ 1230.681676] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 1230.778211] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1230.780074] LNet: Accept secure, port 988 [ 1232.367168] Key type lgssc registered [ 1232.800489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.751272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.758909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.989227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.605629] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1241.950080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1242.277152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.768663] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1245.756203] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1247.113676] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1249.671414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1249.910633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.035383] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1253.932754] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1288.097361] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:01:56 (1753365716) [ 1289.023682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1290.240759] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:01:58 (1753365718) [ 1298.402417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.406203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1298.444612] Lustre: Skipped 2 previous similar messages [ 1298.449490] Lustre: Skipped 1 previous similar message [ 1299.810762] LustreError: 91947:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1299.901567] Lustre: server umount lustre-MDT0000 complete [ 1303.307973] LustreError: 89018:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365731 with bad export cookie 9024798777068212847 [ 1303.328402] LustreError: 89018:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1303.329661] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.474745] LustreError: 92149:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1303.484805] LustreError: 92149:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1303.541371] LustreError: 90647:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.544183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.570060] LustreError: 90647:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1303.771316] Lustre: server umount lustre-MDT0001 complete [ 1313.016647] LustreError: 92350:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1313.028381] LustreError: 92350:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1313.415415] Lustre: server umount lustre-OST0000 complete [ 1323.056577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.387723] LustreError: 92979:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.411730] LustreError: 92979:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1323.535546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.538840] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1328.611818] LustreError: 92979:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.706338] LustreError: 92979:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.158722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1333.683058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.263166] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1340.637514] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.057441] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1353.451584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1362.931299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1363.095917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1363.554940] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1363.560354] LustreError: 92968:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1363.594127] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1363.599990] LustreError: 94829:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -98 [ 1363.621482] LustreError: 94829:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 1363.626249] LustreError: 94829:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1364.003317] Lustre: server umount lustre-OST0000 complete [ 1364.005320] LustreError: 94829:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1376.313418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1376.689966] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1376.712478] LustreError: 92966:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1376.760898] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1376.786594] LustreError: 95295:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -98 [ 1376.807243] LustreError: 95295:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 1376.820434] LustreError: 95295:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1377.098621] Lustre: server umount lustre-OST0000 complete [ 1377.105463] LustreError: 95295:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1382.368532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1391.760418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1392.111297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1392.129218] Lustre: Found index 0 for lustre-OST0000, updating log [ 1392.144619] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1392.282187] Lustre: lustre-OST0000: new disk, initializing [ 1392.284975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1392.700250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.353399] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1403.423505] Lustre: Failing over lustre-OST0000 [ 1403.472110] LustreError: 96379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1403.477170] LustreError: 96379:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1403.607055] Lustre: server umount lustre-OST0000 complete [ 1407.968745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.974186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1407.979923] Lustre: Skipped 1 previous similar message [ 1407.999692] Lustre: Skipped 2 previous similar messages [ 1413.091682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.103439] Lustre: Skipped 1 previous similar message [ 1418.210395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.221894] Lustre: Skipped 1 previous similar message [ 1421.280530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1421.390439] LustreError: 96580:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1421.393754] LustreError: 96580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1421.484731] Lustre: server umount lustre-MDT0000 complete [ 1423.329915] LustreError: 92979:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.345892] LustreError: 92979:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1424.163422] Lustre: 88338:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753365836/real 1753365836] req@0000000021694002 x1838537149340288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753365852 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1424.213479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.005930] LustreError: 96581:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365853 with bad export cookie 9024798777068215577 [ 1425.017955] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.024387] LustreError: 96581:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1425.437203] Lustre: server umount lustre-MDT0001 complete [ 1432.873484] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:04:20 (1753365860) [ 1446.455473] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1454.257868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.613345] LustreError: 98141:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.636864] LustreError: 98141:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1454.729065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.969630] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1465.237542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1469.203864] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1471.821716] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1477.110700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.814961] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1478.828331] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1482.948389] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1490.181977] Lustre: lustre-OST0000: Recovery over after 0:12, of 2 clients 2 recovered and 0 were evicted. [ 1490.183272] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 1490.203039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1490.744918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1490.907655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1490.969416] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1491.082978] Lustre: lustre-OST0001: new disk, initializing [ 1491.092259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1491.376573] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1491.379282] Lustre: Skipped 2 previous similar messages [ 1497.101352] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1497.156158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1497.164877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1497.286694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1504.102231] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1514.088082] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1522.658238] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1522.662739] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.673902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1528.774631] LustreError: 101356:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1528.783843] LustreError: 101356:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1528.871467] Lustre: server umount lustre-MDT0001 complete [ 1530.312286] LustreError: 98137:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.326320] LustreError: 98137:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1531.244939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1537.232555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1537.300838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1537.304470] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1537.335810] Lustre: 98130:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1539.184878] LustreError: 101972:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1539.198408] LustreError: 101972:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1539.216258] LustreError: 98132:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1539.313333] LustreError: 101979:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -17 [ 1539.321360] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1539.321360] [ 1541.014448] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1543.157257] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1543.161507] Lustre: Skipped 1 previous similar message [ 1548.777530] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 10:06:16 (1753365976) [ 1560.663763] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1574.354155] Lustre: 103948:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1593.154908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1596.375500] Lustre: 105082:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1680.864331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1680.872246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.877955] Lustre: Skipped 2 previous similar messages [ 1680.881228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1680.884342] Lustre: Skipped 7 previous similar messages [ 1683.540503] LustreError: 105536:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1683.543600] LustreError: 105536:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1683.593675] Lustre: server umount lustre-MDT0000 complete [ 1685.418698] LustreError: 98120:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366113 with bad export cookie 9024798777068216578 [ 1685.422300] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.425764] LustreError: 98120:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1685.624866] Lustre: server umount lustre-MDT0001 complete [ 1698.146465] Lustre: server umount lustre-OST0000 complete [ 1710.723483] Lustre: server umount lustre-OST0001 complete [ 1713.835238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1718.418041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1718.873953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1722.381100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1725.954156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1726.371342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1733.576260] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1737.631955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.639758] Lustre: lustre-MDT0000: reset Object Index mappings [ 1737.868094] LustreError: 109007:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.880829] LustreError: 109007:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1737.935166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.938934] Lustre: Skipped 1 previous similar message [ 1739.440841] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1742.512444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1742.521043] Lustre: lustre-MDT0001: reset Object Index mappings [ 1744.413668] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1745.764396] Lustre: 110111:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1748.477714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1749.865498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 1751.187704] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1751.914336] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 1754.570664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1756.278122] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 1756.285122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 1757.445862] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1761.365540] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1763.090135] Lustre: 111949:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1784.249422] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:10:12 (1753366212) [ 1814.495875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1814.498616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.503645] Lustre: Skipped 3 previous similar messages [ 1814.506192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1814.509108] Lustre: Skipped 4 previous similar messages [ 1819.338057] LustreError: 112301:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1819.340819] LustreError: 112301:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1819.384031] Lustre: server umount lustre-MDT0000 complete [ 1820.771063] LustreError: 108988:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366249 with bad export cookie 9024798777068263681 [ 1820.774725] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.776772] LustreError: 108988:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1821.664535] LustreError: 110382:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1821.670989] LustreError: 110382:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1827.159087] Lustre: server umount lustre-MDT0001 complete [ 1843.167140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1843.212490] Lustre: server umount lustre-OST0000 complete [ 1850.689697] Lustre: server umount lustre-OST0001 complete [ 1852.815767] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 1856.142921] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1859.829331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1862.135728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1864.291286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1866.528278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1868.825955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1868.852932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.947648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1868.960277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1868.998455] Lustre: lustre-MDT0000: new disk, initializing [ 1869.023711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.027101] Lustre: Skipped 3 previous similar messages [ 1869.034227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1870.289214] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1873.345617] Lustre: server umount lustre-MDT0000 complete [ 1874.323474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1876.590331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1876.614621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.702529] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1876.708363] Lustre: 116148:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1876.715710] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1878.106235] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1879.748511] Lustre: server umount lustre-MDT0000 complete [ 1882.424524] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:11:50 (1753366310) [ 1882.938424] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1883.549870] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:11:52 (1753366312) [ 1888.925477] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1892.323616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.746818] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1896.602961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1896.636597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1896.678622] Lustre: 118150:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1896.707598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1896.710345] Lustre: Skipped 1 previous similar message [ 1896.749815] Lustre: lustre-MDT0001: new disk, initializing [ 1896.813947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1896.818074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1898.207635] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1900.526133] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1902.712516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1902.739182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1902.833835] Lustre: lustre-OST0000: new disk, initializing [ 1902.835501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1903.930926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1903.934786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1903.967913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1904.788753] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1909.035992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.063149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1910.853951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1910.857137] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1910.901763] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1911.059522] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1915.407408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1916.558179] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2121.861443] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:15:50 (1753366550) [ 2135.423111] LustreError: 127091:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2135.425395] LustreError: 127091:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 2135.486187] Lustre: server umount lustre-MDT0000 complete [ 2136.545693] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.548022] LustreError: 121493:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.550354] Lustre: Skipped 8 previous similar messages [ 2136.554178] LustreError: 121493:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2136.693109] LustreError: 126899:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366565 with bad export cookie 9024798777068317959 [ 2136.695605] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.696081] LustreError: 126899:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2141.665766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2141.667465] Lustre: Skipped 9 previous similar messages [ 2142.965882] Lustre: server umount lustre-MDT0001 complete [ 2150.337341] Lustre: server umount lustre-OST0000 complete [ 2157.676733] Lustre: server umount lustre-OST0001 complete [ 2159.723637] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 2162.758752] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2166.300147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2168.421869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2170.415755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.424136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2174.581225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2174.602707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.684202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2174.696684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2174.698378] Lustre: Skipped 1 previous similar message [ 2174.733362] Lustre: lustre-MDT0000: new disk, initializing [ 2174.735086] Lustre: Skipped 1 previous similar message [ 2174.761868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.764833] Lustre: Skipped 5 previous similar messages [ 2174.771737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2176.064747] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2180.120974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2180.144446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2180.181958] Lustre: 129858:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2180.286760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2181.553811] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2184.641923] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2185.779980] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2187.930321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.953572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2189.690026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2189.725303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2189.768084] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2192.865903] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2196.473854] Lustre: lustre-MDD0000: changelog on [ 2197.412301] Lustre: lustre-MDD0001: changelog on [ 2912.468749] Lustre: lustre-MDD0001: changelog off [ 2913.180938] Lustre: 134273:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2913.535576] Lustre: lustre-MDD0000: changelog off [ 2916.833894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2916.834731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.837818] LustreError: Skipped 1 previous similar message [ 2916.838742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.843259] Lustre: Skipped 3 previous similar messages [ 2921.390176] LustreError: 139339:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2921.392756] LustreError: 139339:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2921.472772] Lustre: server umount lustre-OST0000 complete [ 2922.975846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2927.072846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.072969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2927.078283] Lustre: Skipped 1 previous similar message [ 2927.080813] Lustre: Skipped 2 previous similar messages [ 2937.311113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2937.418467] Lustre: server umount lustre-MDT0000 complete [ 2938.731513] LustreError: 129850:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367367 with bad export cookie 9024798777069122133 [ 2938.736378] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2938.736512] LustreError: 129850:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2938.803056] LustreError: 139742:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2938.805978] LustreError: 139742:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2938.896882] Lustre: server umount lustre-MDT0001 complete [ 2943.834606] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2944.897040] Key type lgssc unregistered [ 2945.033519] LNet: 140270:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2946.085306] LNet: Removed LNI 192.168.201.123@tcp [ 2946.440781] Key type .llcrypt unregistered [ 2946.442364] Key type ._llcrypt unregistered [ 2953.311284] Key type ._llcrypt registered [ 2953.313036] Key type .llcrypt registered [ 2953.351111] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 2959.632542] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2959.980627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.028060] alg: No test for adler32 (adler32-zlib) [ 2960.905986] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2960.996525] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2960.998704] LNet: Accept secure, port 988 [ 2962.591159] Key type lgssc registered [ 2962.985226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2966.407934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2968.853079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.145850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2973.476071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2976.184677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2976.204680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2976.210878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.312704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2977.327334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2977.369212] Lustre: lustre-MDT0000: new disk, initializing [ 2977.400402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.409264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2978.942593] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2983.677922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2983.704437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2983.747320] Lustre: 143410:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2983.779432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2983.782021] Lustre: Skipped 1 previous similar message [ 2983.820930] Lustre: lustre-MDT0001: new disk, initializing [ 2983.862900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2983.877769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2983.882450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2985.401226] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2988.722690] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.999480] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2992.532383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.563848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2992.667900] Lustre: lustre-OST0000: new disk, initializing [ 2992.671134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2992.708108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.801191] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2995.132692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2995.136861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2995.175211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2998.078868] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3001.428248] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_module obdecho/obdecho [ 3002.107215] LustreError: 146005:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3002.112028] LustreError: 146005:0:(obd_config.c:779:class_setup()) setup ec failed (-22) [ 3002.751393] LustreError: 146101:0:(echo_client.c:1878:echo_md_handler()) Can not resolve the path : rc = -2 [ 3009.504030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3009.507190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.512685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.528381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.530119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.531396] Lustre: Skipped 1 previous similar message [ 3010.534928] Lustre: Skipped 1 previous similar message [ 3012.413493] LustreError: 146441:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3012.445931] Lustre: server umount lustre-MDT0000 complete [ 3013.796542] LustreError: 143402:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367442 with bad export cookie 5856483799722624677 [ 3013.801307] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3013.869818] LustreError: 146644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3013.873204] LustreError: 146644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3013.964031] Lustre: server umount lustre-MDT0001 complete [ 3025.585306] LustreError: 146845:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3025.587691] LustreError: 146845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3025.672923] Lustre: server umount lustre-OST0000 complete [ 3033.583870] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3037.479948] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3041.295092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3043.872906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3046.362969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.036267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.667198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.695936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.781278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3051.794848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3051.835531] Lustre: lustre-MDT0000: new disk, initializing [ 3051.862550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.871145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3053.381396] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3057.904334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.930828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3057.971046] Lustre: 149771:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3057.999296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3058.002308] Lustre: Skipped 1 previous similar message [ 3058.042780] Lustre: lustre-MDT0001: new disk, initializing [ 3058.080361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.095471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3058.100816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3059.597666] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3062.918449] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3064.323179] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3067.000418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3067.026478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3067.135194] Lustre: lustre-OST0000: new disk, initializing [ 3067.137735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3067.170665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.212891] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3069.687779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3069.691533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3069.725918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3072.514820] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3074.041278] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3074.125691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3075.635634] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3075.729842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3078.508078] LustreError: 152674:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3078.511052] LustreError: 152674:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3078.580809] Lustre: server umount lustre-OST0000 complete [ 3080.160604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.161534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.165996] Lustre: Skipped 1 previous similar message [ 3080.168843] Lustre: Skipped 1 previous similar message [ 3085.281287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.284440] Lustre: Skipped 1 previous similar message [ 3090.400579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3090.404892] LustreError: 152875:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3090.406895] Lustre: Skipped 1 previous similar message [ 3090.409923] LustreError: 152875:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3090.442415] Lustre: server umount lustre-MDT0000 complete [ 3092.096233] LustreError: 149763:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367520 with bad export cookie 5856483799722625622 [ 3092.096684] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.101863] LustreError: 149763:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3102.690994] LustreError: 153078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3102.693840] LustreError: 153078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3102.785417] Lustre: server umount lustre-MDT0001 complete [ 3104.155384] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:32:12 (1753367532) [ 3104.766437] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3105.482363] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:32:13 (1753367533) [ 3108.428086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.663925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.089135] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3113.552699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3115.328491] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3117.276455] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3118.649285] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3121.176882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.395236] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3125.416247] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3139.043131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.044410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.049586] Lustre: Skipped 2 previous similar messages [ 3139.052204] Lustre: Skipped 1 previous similar message [ 3144.451267] LustreError: 155999:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3144.454466] LustreError: 155999:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3144.488136] Lustre: server umount lustre-MDT0000 complete [ 3145.889184] LustreError: 153730:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367574 with bad export cookie 5856483799722626490 [ 3145.892306] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3145.894447] LustreError: 153730:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3146.181093] LustreError: 153745:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.188512] LustreError: 153745:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3149.280048] LustreError: 153745:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.280354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.303897] LustreError: 153745:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.085514] Lustre: server umount lustre-MDT0001 complete [ 3154.506821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3159.311048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3159.411790] LustreError: 157257:0:(llog.c:1602:llog_backup()) MGC192.168.201.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3159.415033] Lustre: 157257:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.201.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3174.815909] LustreError: 157265:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.823099] LustreError: 157265:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3179.936185] LustreError: 157265:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.943378] LustreError: 157265:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3190.239736] LustreError: 157264:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.244914] LustreError: 157264:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [ 3207.620800] LustreError: 157265:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.628071] LustreError: 157265:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [ 3220.063333] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3220.066041] Lustre: 157257:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.201.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3240.928460] LustreError: 157281:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.934920] LustreError: 157281:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 21 previous similar messages [ 3248.607313] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3248.610116] Lustre: 153740:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.201.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3260.902510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3260.905716] Lustre: Skipped 2 previous similar messages [ 3262.209572] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3265.058813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.019186] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 3266.022528] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0x0 to 0x514668fca34c26fe [ 3267.416308] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3269.316971] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3269.914084] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3271.132119] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3271.144149] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3271.146490] Lustre: Skipped 1 previous similar message [ 3273.774555] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3279.693421] LustreError: 158729:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3279.696651] LustreError: 158729:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3279.726565] Lustre: server umount lustre-MDT0000 complete [ 3281.278707] LustreError: 153729:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367709 with bad export cookie 5856483799722632958 [ 3281.283414] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.285262] LustreError: 153729:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3281.377039] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.383838] Lustre: Skipped 1 previous similar message [ 3281.462559] Lustre: server umount lustre-MDT0001 complete [ 3285.670349] Lustre: server umount lustre-OST0000 complete [ 3302.215833] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3309.754235] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3318.026928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3324.037709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3328.837043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3334.743489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3342.401933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3342.493266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.795878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3342.845226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3342.991371] Lustre: lustre-MDT0000: new disk, initializing [ 3343.121949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.135833] Lustre: Skipped 1 previous similar message [ 3343.168069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3347.385947] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3357.740336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.800348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3357.859762] Lustre: 162059:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3358.064198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3361.186043] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3366.982960] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3369.340627] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3374.045712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3374.099435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3376.606860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3376.611020] Lustre: Skipped 1 previous similar message [ 3376.613610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3376.685749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3378.061437] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3383.051781] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3385.533136] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3385.647844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3387.658237] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3387.775339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3391.970578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3391.971425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.972781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3391.972786] Lustre: Skipped 8 previous similar messages [ 3391.980022] LustreError: Skipped 1 previous similar message [ 3391.992493] Lustre: Skipped 1 previous similar message [ 3403.231137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.257456] LustreError: 164967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3403.260178] LustreError: 164967:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3403.357568] Lustre: server umount lustre-OST0000 complete [ 3411.772079] Lustre: server umount lustre-MDT0000 complete [ 3412.448762] LustreError: 162071:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.461937] LustreError: 162071:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 20 previous similar messages [ 3413.963896] LustreError: 163804:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367842 with bad export cookie 5856483799722649737 [ 3413.966614] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.969737] LustreError: 163804:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3414.172065] Lustre: server umount lustre-MDT0001 complete [ 3419.802988] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:37:27 (1753367847) [ 3424.253032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.219525] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3432.858921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3435.468493] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3438.773476] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3441.005437] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3445.168789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3448.532466] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3451.481346] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3466.336759] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:38:14 (1753367894) [ 3468.256789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.258801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.260555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3468.260560] Lustre: Skipped 7 previous similar messages [ 3468.262952] Lustre: Skipped 3 previous similar messages [ 3474.366716] Lustre: server umount lustre-OST0000 complete [ 3482.320509] Lustre: server umount lustre-MDT0000 complete [ 3484.854859] LustreError: 166020:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367913 with bad export cookie 5856483799722650626 [ 3484.860659] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3484.866632] LustreError: 166020:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3485.113311] Lustre: server umount lustre-MDT0001 complete [ 3491.661753] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3493.337650] Key type lgssc unregistered [ 3493.538075] LNet: 169415:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3494.565853] LNet: Removed LNI 192.168.201.123@tcp [ 3495.025617] Key type .llcrypt unregistered [ 3495.027106] Key type ._llcrypt unregistered [ 3504.805030] Key type ._llcrypt registered [ 3504.806696] Key type .llcrypt registered [ 3504.882544] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3505.359400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.482644] alg: No test for adler32 (adler32-zlib) [ 3506.437108] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 3506.592157] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3506.595895] LNet: Accept secure, port 988 [ 3508.232470] Key type lgssc registered [ 3508.928103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3513.729054] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3513.765027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3530.271674] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3536.415581] LustreError: 170603:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 3542.559650] LustreError: 170603:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3542.566760] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3542.571765] LustreError: MGC192.168.201.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3542.577819] LustreError: 170603:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3542.581825] LustreError: 170603:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -22 [ 3542.586244] LustreError: 170603:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 3542.592149] LustreError: 170603:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3542.685262] Lustre: server umount lustre-OST0000 complete [ 3542.689953] LustreError: 170603:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3548.266908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.561879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.069242] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3557.577565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3557.795193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3560.110490] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3563.612181] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3565.785933] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3570.236613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3570.388604] LustreError: 172683:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3570.394562] Lustre: 172683:0:(mgc_request_server.c:610:mgc_llog_local_copy()) MGC192.168.201.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3570.505292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.530810] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3578.489913] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.015215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.026644] Lustre: Skipped 1 previous similar message [ 3583.027341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3583.040864] Lustre: Skipped 2 previous similar messages [ 3588.064583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.068127] Lustre: Skipped 2 previous similar messages [ 3593.186682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.189695] Lustre: Skipped 2 previous similar messages [ 3594.207190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3594.258359] LustreError: 173330:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3594.261656] LustreError: 173330:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3594.314125] Lustre: server umount lustre-MDT0000 complete [ 3596.717464] LustreError: 173331:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368025 with bad export cookie 6252749004588326484 [ 3596.724615] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3596.727278] LustreError: 173331:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3597.407829] LustreError: 171814:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e915559b x1838539534787584/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3597.435847] LustreError: 173534:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3597.441831] LustreError: 173534:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3597.564681] Lustre: server umount lustre-MDT0001 complete [ 3610.245363] LustreError: 173736:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3610.249423] LustreError: 173736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3610.359507] Lustre: server umount lustre-OST0000 complete [ 3615.657312] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3621.208423] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3626.518956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.808634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3632.704632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3636.067761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.824109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.856990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.980833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3640.003555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3640.047903] Lustre: lustre-MDT0000: new disk, initializing [ 3640.102382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.115652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3642.173262] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3648.624887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3648.688627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3648.748073] Lustre: 176055:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3648.793516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3648.796078] Lustre: Skipped 1 previous similar message [ 3648.840601] Lustre: lustre-MDT0001: new disk, initializing [ 3648.905885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.923701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3648.929541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3651.102147] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3655.499705] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3657.360646] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3661.161233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3661.207876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3661.414630] Lustre: lustre-OST0000: new disk, initializing [ 3661.418496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3661.473470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.060536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3663.065675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3663.142017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3664.837561] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3668.982445] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3671.018806] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3671.141706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3673.169813] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3673.290847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3678.693380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3678.694701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.697386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.705233] Lustre: Skipped 2 previous similar messages [ 3680.302285] LustreError: 178958:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3680.306441] LustreError: 178958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3680.386282] Lustre: server umount lustre-OST0000 complete [ 3683.810397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.811297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.816254] Lustre: Skipped 1 previous similar message [ 3683.819351] Lustre: Skipped 1 previous similar message [ 3688.716356] LustreError: 179159:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3688.719171] LustreError: 179159:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3688.754088] Lustre: server umount lustre-MDT0000 complete [ 3688.928101] LustreError: 176842:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.937074] LustreError: 176842:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3690.756290] LustreError: 176045:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368119 with bad export cookie 6252749004588327205 [ 3690.759648] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.762201] LustreError: 176045:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3690.945284] Lustre: server umount lustre-MDT0001 complete [ 3694.943805] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:42:03 (1753368123) [ 3702.538111] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3704.008465] Key type lgssc unregistered [ 3704.162690] LNet: 180545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.190823] LNet: Removed LNI 192.168.201.123@tcp [ 3705.635459] Key type .llcrypt unregistered [ 3705.637465] Key type ._llcrypt unregistered [ 3714.624214] Key type ._llcrypt registered [ 3714.626281] Key type .llcrypt registered [ 3714.679565] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3715.189045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3715.204984] alg: No test for adler32 (adler32-zlib) [ 3716.128586] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 3716.238591] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3716.242182] LNet: Accept secure, port 988 [ 3717.847209] Key type lgssc registered [ 3718.425914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3722.553924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3722.569389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.820551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.794269] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3730.698507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3730.910351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.007967] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3736.174273] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3738.193224] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3742.188686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3742.319833] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3742.322114] LustreError: 183336:0:(llog.c:1642:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3742.326580] Lustre: 183336:0:(mgc_request_server.c:607:mgc_llog_local_copy()) MGC192.168.201.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3742.383743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.438188] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3748.229877] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3751.392092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3751.396223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.405276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.928496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.935556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.597310] LustreError: 183978:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3755.660554] Lustre: server umount lustre-MDT0000 complete [ 3756.002968] LustreError: 182748:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.015818] LustreError: 182748:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3757.703273] LustreError: 181692:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368186 with bad export cookie 14895416110547705627 [ 3757.708804] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3757.710518] LustreError: 181692:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3758.055260] LustreError: 182418:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003a1f7933 x1838539754983168/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3758.077027] LustreError: 184182:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3758.080340] LustreError: 184182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3758.209496] Lustre: server umount lustre-MDT0001 complete [ 3770.547439] LustreError: 184385:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3770.550710] LustreError: 184385:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3770.643170] Lustre: server umount lustre-OST0000 complete [ 3774.930528] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3779.651661] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3784.269824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3787.725437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3791.197448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3794.788283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3798.104907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3798.135350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.235786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3798.253162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3798.293455] Lustre: lustre-MDT0000: new disk, initializing [ 3798.326847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.336881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3799.979978] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3805.249982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3805.300580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3805.347570] Lustre: 186706:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3805.376670] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3805.379339] Lustre: Skipped 1 previous similar message [ 3805.381463] LustreError: 187464:0:(llog.c:1642:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3805.386088] LustreError: 187464:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3805.390130] Lustre: 187464:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.201.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3805.401402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3805.404222] Lustre: Skipped 1 previous similar message [ 3805.446459] Lustre: lustre-MDT0001: new disk, initializing [ 3805.500351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3805.521862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3805.528242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3807.236385] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3810.966522] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3812.514375] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3815.400931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3815.425120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3815.532033] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3815.534356] Lustre: Skipped 1 previous similar message [ 3815.536514] LustreError: 188415:0:(llog.c:1642:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3815.540862] LustreError: 188415:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3815.543717] Lustre: 188415:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.201.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3815.548281] Lustre: 188415:0:(mgc_request_server.c:630:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3815.560979] Lustre: lustre-OST0000: new disk, initializing [ 3815.563865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3815.604350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3818.075807] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3818.637466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3818.642058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3818.697264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3821.955278] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3823.685080] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3823.773045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3825.453320] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3825.560572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3829.219440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.222523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.225311] LustreError: Skipped 1 previous similar message [ 3829.230465] Lustre: Skipped 1 previous similar message [ 3829.234837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.241391] Lustre: Skipped 3 previous similar messages [ 3832.502115] LustreError: 189610:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3832.504421] LustreError: 189610:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3832.591436] Lustre: server umount lustre-OST0000 complete [ 3836.384515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3836.388103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.396852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.525315] LustreError: 189811:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3840.529155] LustreError: 189811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3840.578537] Lustre: server umount lustre-MDT0000 complete [ 3842.353197] LustreError: 188439:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368270 with bad export cookie 14895416110547706313 [ 3842.354872] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.358225] LustreError: 188439:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3842.585892] Lustre: server umount lustre-MDT0001 complete [ 3846.300493] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:44:34 (1753368274) [ 3852.725568] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3857.107682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.316023] LustreError: 191367:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.341144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.241778] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3862.496299] LustreError: 191367:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.522442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3865.509904] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3866.759823] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3869.851799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3872.582692] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3876.458516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3876.490953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3876.563379] Lustre: lustre-OST0001: new disk, initializing [ 3876.566288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3878.943262] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3881.492514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3881.497106] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3881.536774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3889.093845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3897.204781] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3903.005607] Lustre: DEBUG MARKER: ADD OST3 [ 3905.136210] loop: module loaded [ 3905.160254] loop0: detected capacity change from 0 to 204800000 [ 3905.252173] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3905.258174] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3905.271317] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3905.341920] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3908.174766] loop0: detected capacity change from 0 to 204800000 [ 3910.619310] Lustre: DEBUG MARKER: STOP OST3 [ 3911.036316] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3911.083077] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3911.187234] Lustre: lustre-OST0003: new disk, initializing [ 3911.190071] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3911.253874] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3911.257714] Lustre: Skipped 3 previous similar messages [ 3912.846078] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3912.910885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3912.915571] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3912.941203] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3913.833425] Lustre: Failing over lustre-OST0003 [ 3913.877824] LustreError: 195865:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3913.881202] LustreError: 195865:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3913.917128] Lustre: server umount lustre-OST0003 complete [ 3914.624335] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3914.911088] LustreError: 195691:0:(osp_precreate.c:1283:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3916.502587] LustreError: 192807:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.314867] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.320376] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.326650] Lustre: Skipped 2 previous similar messages [ 3921.175248] Lustre: DEBUG MARKER: START OST3 again [ 3921.866248] LustreError: 192806:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.881626] LustreError: 192806:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3924.786319] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3924.908368] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3926.238155] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3926.723952] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3926.723959] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 3927.768801] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3943.686091] Lustre: Failing over lustre-OST0003 [ 3943.715130] LustreError: 197771:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3943.718416] LustreError: 197771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3943.741191] Lustre: server umount lustre-OST0003 complete [ 3945.449089] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.457170] LustreError: 192808:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.462049] Lustre: Skipped 1 previous similar message [ 3945.473129] LustreError: 192808:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3960.806284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.809720] Lustre: Skipped 4 previous similar messages [ 3963.910497] Lustre: server umount lustre-MDT0000 complete [ 3965.922634] LustreError: 192825:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.933693] LustreError: 192825:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [ 3965.979172] LustreError: 191349:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368394 with bad export cookie 14895416110547707195 [ 3965.984786] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.986365] LustreError: 191349:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3971.044914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.049475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3971.051515] Lustre: Skipped 5 previous similar messages [ 3971.057468] Lustre: Skipped 1 previous similar message [ 3972.279904] Lustre: server umount lustre-MDT0001 complete [ 3974.002872] Lustre: server umount lustre-OST0000 complete [ 3975.662870] Lustre: server umount lustre-OST0001 complete [ 3978.595411] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3982.466500] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3988.111578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3991.643050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3995.042287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3998.533108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4001.991610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4002.022213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.137262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4002.158438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4002.201094] Lustre: lustre-MDT0000: new disk, initializing [ 4002.246600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.249840] Lustre: Skipped 1 previous similar message [ 4002.257360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4004.104205] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4010.216396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4010.272165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4010.335313] Lustre: 200812:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4010.497993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4012.448218] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4016.909996] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4018.881171] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4022.509480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4022.554411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4022.700248] Lustre: lustre-OST0000: new disk, initializing [ 4022.703377] Lustre: Skipped 1 previous similar message [ 4022.707940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4022.710517] Lustre: Skipped 2 previous similar messages [ 4024.154052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4024.158063] Lustre: Skipped 1 previous similar message [ 4024.160875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4024.218293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4025.668715] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4030.090121] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4032.169461] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4032.271772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4034.178398] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4034.294975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4039.651363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.651473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4039.659524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.661284] LustreError: Skipped 1 previous similar message [ 4039.663140] Lustre: Skipped 1 previous similar message [ 4041.329129] LustreError: 203718:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4041.332368] LustreError: 203718:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 4041.416693] Lustre: server umount lustre-OST0000 complete [ 4049.241521] Lustre: server umount lustre-MDT0000 complete [ 4049.888441] LustreError: 200823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.895307] LustreError: 200823:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4050.899170] LustreError: 200804:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368479 with bad export cookie 14895416110547709463 [ 4050.906089] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4051.086718] Lustre: server umount lustre-MDT0001 complete [ 4052.781300] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:48:01 (1753368481) [ 4059.455305] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4063.610535] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4068.077844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4070.826707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.429499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4076.320429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4079.367493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4079.399242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.498487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4079.516332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4079.556740] Lustre: lustre-MDT0000: new disk, initializing [ 4079.594773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4081.260116] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4086.935072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4086.969860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4087.017617] Lustre: 206944:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4087.149196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4088.974365] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4092.616946] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4094.119990] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4097.178195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4097.217082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4098.807892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4099.885466] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4103.710096] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4105.608697] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.717077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4107.867305] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4108.024114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4109.280498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.283879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.284739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4109.284743] Lustre: Skipped 2 previous similar messages [ 4109.294597] Lustre: Skipped 4 previous similar messages [ 4115.116662] LustreError: 209853:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4115.119066] LustreError: 209853:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4115.209334] Lustre: server umount lustre-OST0000 complete [ 4131.295458] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4131.441597] Lustre: server umount lustre-MDT0000 complete [ 4133.262661] LustreError: 207718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368561 with bad export cookie 14895416110547710345 [ 4133.265909] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.271343] LustreError: 207718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4133.441800] Lustre: server umount lustre-MDT0001 complete [ 4137.299210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.474555] LustreError: 210731:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.501474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.505488] Lustre: Skipped 5 previous similar messages [ 4139.093665] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4142.663585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4144.437311] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4146.753602] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.551291] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4151.593128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4153.983638] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4156.204269] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4172.258951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4172.262602] LustreError: Skipped 2 previous similar messages [ 4174.640114] Lustre: server umount lustre-OST0000 complete [ 4177.378827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.381544] Lustre: Skipped 13 previous similar messages [ 4182.363493] Lustre: server umount lustre-MDT0000 complete [ 4183.951540] LustreError: 210712:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368612 with bad export cookie 14895416110547711227 [ 4183.953248] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4183.956194] LustreError: 210712:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4184.103767] Lustre: server umount lustre-MDT0001 complete [ 4290.804874] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 4292.027851] Key type lgssc unregistered [ 4292.175940] LNet: 214038:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4293.221856] LNet: Removed LNI 192.168.201.123@tcp [ 4293.610808] Key type .llcrypt unregistered [ 4293.612909] Key type ._llcrypt unregistered [ 4309.761690] Key type ._llcrypt registered [ 4309.776727] Key type .llcrypt registered [ 4309.918978] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4325.439727] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4326.822785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4327.044928] alg: No test for adler32 (adler32-zlib) [ 4328.370824] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 4328.686458] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 4328.692949] LNet: Accept secure, port 988 [ 4330.383256] Key type lgssc registered [ 4331.945438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4344.669175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.865464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4357.332378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4361.709991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.572024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.608366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4366.623104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.802423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4367.824888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4367.899646] Lustre: lustre-MDT0000: new disk, initializing [ 4367.977780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.005064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4370.641423] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4377.890713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4377.944972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4377.994375] Lustre: 217350:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4378.053406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4378.056563] Lustre: Skipped 1 previous similar message [ 4378.100582] Lustre: lustre-MDT0001: new disk, initializing [ 4378.166409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.196849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4378.204852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4380.184482] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4384.761520] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4386.906513] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4390.615766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4390.659838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4390.834720] Lustre: lustre-OST0000: new disk, initializing [ 4390.838540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4390.890370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.847387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4392.857070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4392.905183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4394.176240] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4398.989218] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4401.527328] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4401.665189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4404.313721] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4404.524143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4408.294446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4408.296799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.305229] LustreError: Skipped 1 previous similar message [ 4408.306489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.319233] Lustre: Skipped 1 previous similar message [ 4413.410919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4413.417829] Lustre: Skipped 2 previous similar messages [ 4418.530305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4418.534631] Lustre: Skipped 1 previous similar message [ 4420.063554] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4420.091707] LustreError: 220257:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4420.211029] Lustre: server umount lustre-OST0000 complete [ 4423.650543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.652170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.657167] Lustre: Skipped 1 previous similar message [ 4428.619098] LustreError: 220459:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4428.622560] LustreError: 220459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4428.668931] Lustre: server umount lustre-MDT0000 complete [ 4430.800962] LustreError: 217341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368859 with bad export cookie 17238000451984399435 [ 4430.801500] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4430.810287] LustreError: 217341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4430.900716] LustreError: 220662:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4430.907451] LustreError: 220662:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4431.056143] Lustre: server umount lustre-MDT0001 complete [ 4440.922312] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4446.981495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.247266] LustreError: 221828:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.259359] LustreError: 221828:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4447.284672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.738215] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4452.320966] LustreError: 221827:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.482719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.728035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4456.846618] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4458.404505] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4461.590189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4464.480114] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4469.409918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4469.468709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4469.571469] Lustre: lustre-OST0001: new disk, initializing [ 4469.575631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4469.664749] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4469.667721] Lustre: Skipped 1 previous similar message [ 4471.728950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4471.733188] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4471.791798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4473.358180] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4480.658975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4484.164142] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4497.234989] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:55:25 (1753368925) [ 4500.963380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4500.966531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.975333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4500.979419] Lustre: Skipped 1 previous similar message [ 4504.882189] LustreError: 225186:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4504.887827] LustreError: 225186:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4504.984916] Lustre: server umount lustre-MDT0000 complete [ 4507.402838] LustreError: 224823:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368935 with bad export cookie 17238000451984400317 [ 4507.407273] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4507.418163] LustreError: 224823:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4507.695656] Lustre: server umount lustre-MDT0001 complete [ 4520.188179] LustreError: 225587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4520.196814] LustreError: 225587:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4520.263409] Lustre: server umount lustre-OST0000 complete [ 4532.419528] LustreError: 225789:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4532.422862] LustreError: 225789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4532.527494] Lustre: server umount lustre-OST0001 complete [ 4535.840252] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4540.851163] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4546.182719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4550.100661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.603299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4557.539240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.463399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.511710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.666402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4561.695348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4561.781979] Lustre: lustre-MDT0000: new disk, initializing [ 4561.874246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.889281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4564.214986] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4571.085579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.151511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4571.219269] Lustre: 227959:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4571.262877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4571.265900] Lustre: Skipped 1 previous similar message [ 4571.317281] Lustre: lustre-MDT0001: new disk, initializing [ 4571.424753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4571.430681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4574.072423] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4579.447472] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4581.766332] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4586.177408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.232725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4587.864293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4587.869061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4587.947254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4589.793364] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4594.390578] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4596.578903] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4596.673836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4598.858173] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4598.977411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4603.360031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4603.361571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.364262] LustreError: Skipped 1 previous similar message [ 4603.366658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.370418] Lustre: Skipped 4 previous similar messages [ 4603.372883] Lustre: Skipped 4 previous similar messages [ 4606.255125] LustreError: 230864:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4606.257630] LustreError: 230864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4606.343735] Lustre: server umount lustre-OST0000 complete [ 4608.482792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.582037] Lustre: server umount lustre-MDT0000 complete [ 4616.536828] LustreError: 227950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369044 with bad export cookie 17238000451984402459 [ 4616.540698] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.542786] LustreError: 227950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4616.731515] Lustre: server umount lustre-MDT0001 complete [ 4621.291824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.570831] LustreError: 231744:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.636990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4621.640818] Lustre: Skipped 2 previous similar messages [ 4623.932925] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4626.914706] LustreError: 231744:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.984983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4631.364445] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4634.530450] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.407559] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4639.876614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4642.826618] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4645.430932] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4792.802662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4792.802698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.806693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4792.811935] Lustre: Skipped 2 previous similar messages [ 4792.816731] Lustre: Skipped 6 previous similar messages [ 4806.111468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4806.127323] LustreError: 234010:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4806.129578] LustreError: 234010:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4806.223978] Lustre: server umount lustre-OST0000 complete [ 4808.672920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.290532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.297319] Lustre: Skipped 1 previous similar message [ 4814.375227] Lustre: server umount lustre-MDT0000 complete [ 4816.317466] LustreError: 231724:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369244 with bad export cookie 17238000451984403341 [ 4816.319291] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.322377] LustreError: 231724:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4816.523745] Lustre: server umount lustre-MDT0001 complete [ 4909.488373] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 4910.809814] Key type lgssc unregistered [ 4910.965483] LNet: 234987:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4912.037150] LNet: Removed LNI 192.168.201.123@tcp [ 4912.514660] Key type .llcrypt unregistered [ 4912.516502] Key type ._llcrypt unregistered [ 4924.647403] Key type ._llcrypt registered [ 4924.649122] Key type .llcrypt registered [ 4924.711902] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4932.536973] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4932.936364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4932.976040] alg: No test for adler32 (adler32-zlib) [ 4933.900076] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 4934.021497] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 4934.025875] LNet: Accept secure, port 988 [ 4935.623285] Key type lgssc registered [ 4936.099176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4940.249404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4943.278439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4946.125620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4949.443745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.443648] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4962.532285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4962.560476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4962.576553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4963.719452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4963.741273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4963.790474] Lustre: lustre-MDT0000: new disk, initializing [ 4963.844162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4963.853541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4965.956341] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4972.209657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4972.245367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4972.307633] Lustre: 238961:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4972.341346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4972.344125] Lustre: Skipped 1 previous similar message [ 4972.385290] Lustre: lustre-MDT0001: new disk, initializing [ 4972.443451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4972.472120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4972.480424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4974.261166] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4977.265863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4980.825433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4980.858161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4981.008407] Lustre: lustre-OST0000: new disk, initializing [ 4981.016392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4981.077378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.961077] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4984.909857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4984.914180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4984.968289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4989.887955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4989.939400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4990.016957] Lustre: lustre-OST0001: new disk, initializing [ 4990.020514] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4990.076599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4993.000851] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4993.079174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4993.086620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4993.165496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5001.371646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5006.198756] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5018.257331] LustreError: 242457:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5018.297478] Lustre: server umount lustre-MDT0000 complete [ 5018.593439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5018.598249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.109330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.117952] Lustre: Skipped 2 previous similar messages [ 5020.111725] LustreError: 241403:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369448 with bad export cookie 9607511815121408293 [ 5020.114481] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.118635] LustreError: 241403:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5022.216746] LustreError: 242658:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5022.220019] LustreError: 242658:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5022.330996] Lustre: server umount lustre-MDT0001 complete [ 5034.490241] LustreError: 242858:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5034.493548] LustreError: 242858:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5034.521960] Lustre: server umount lustre-OST0000 complete [ 5046.580663] LustreError: 243060:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5046.584079] LustreError: 243060:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5046.679455] Lustre: server umount lustre-OST0001 complete [ 5052.737795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5056.567198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5057.001786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5060.495546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5064.370157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5064.846378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5072.318816] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5076.252653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.262503] Lustre: lustre-MDT0000: reset Object Index mappings [ 5076.443871] LustreError: 246357:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.451459] LustreError: 246357:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 5076.466613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.052490] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5081.568733] LustreError: 246357:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.965500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.974437] Lustre: lustre-MDT0001: reset Object Index mappings [ 5083.829861] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5085.204983] Lustre: 247456:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5088.492025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5088.699802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.704731] Lustre: Skipped 1 previous similar message [ 5091.293431] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5091.751646] LustreError: 248217:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.014482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5097.799741] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5100.714351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5100.715365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5107.013710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5108.840493] Lustre: 249288:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5116.388965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.392516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.395945] Lustre: Skipped 3 previous similar messages [ 5116.401402] Lustre: Skipped 3 previous similar messages [ 5121.505193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.508167] Lustre: Skipped 2 previous similar messages [ 5121.618521] LustreError: 249391:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5121.622709] LustreError: 249391:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5121.677134] Lustre: server umount lustre-MDT0000 complete [ 5123.370061] LustreError: 246337:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369551 with bad export cookie 9607511815121415083 [ 5123.374426] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5123.377922] LustreError: 246337:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5123.560596] Lustre: server umount lustre-MDT0001 complete [ 5134.842479] LustreError: 249794:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5134.845816] LustreError: 249794:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5134.872790] Lustre: server umount lustre-OST0000 complete [ 5147.162439] Lustre: server umount lustre-OST0001 complete [ 5156.127669] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:06:24 (1753369584) [ 5164.136724] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 5168.120094] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5172.630773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5175.652578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.332088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5181.187043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5184.428617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5184.463654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.582968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5184.605412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5184.646526] Lustre: lustre-MDT0000: new disk, initializing [ 5184.688052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.691212] Lustre: Skipped 1 previous similar message [ 5184.700568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5186.553568] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5191.981641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5192.032956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5192.081183] Lustre: 253499:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5192.207233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5193.756597] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5197.555138] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5199.311791] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5202.704626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.740303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5202.870250] Lustre: lustre-OST0000: new disk, initializing [ 5202.872607] Lustre: Skipped 1 previous similar message [ 5202.875902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5202.879072] Lustre: Skipped 2 previous similar messages [ 5204.548077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5204.555247] Lustre: Skipped 1 previous similar message [ 5204.559794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5204.600585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5205.596433] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5209.951625] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5211.946145] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5212.065165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5213.767416] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5213.868973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5215.203687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5215.207768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.209124] LustreError: Skipped 1 previous similar message [ 5215.209682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5215.214066] Lustre: Skipped 1 previous similar message [ 5215.220553] Lustre: Skipped 1 previous similar message [ 5220.324828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5220.327345] Lustre: Skipped 1 previous similar message [ 5220.723289] LustreError: 256407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5220.727516] LustreError: 256407:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5220.807957] Lustre: server umount lustre-OST0000 complete [ 5222.880165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5222.885369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.440397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5225.443421] Lustre: Skipped 2 previous similar messages [ 5228.524838] Lustre: server umount lustre-MDT0000 complete [ 5230.383480] LustreError: 253490:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369658 with bad export cookie 9607511815121417176 [ 5230.387024] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5230.391548] LustreError: 253490:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5230.598244] Lustre: server umount lustre-MDT0001 complete [ 5239.263295] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5244.060208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.284688] LustreError: 257974:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.320593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.323507] Lustre: Skipped 2 previous similar messages [ 5246.198323] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5249.504388] LustreError: 257974:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5250.130896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5252.142602] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5253.477442] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5256.773915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5259.500983] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5263.843233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5263.876396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5263.964911] Lustre: lustre-OST0001: new disk, initializing [ 5263.968291] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5265.562604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5265.567594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5265.615714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5266.732679] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5272.504491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5274.254084] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5281.698121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5281.700763] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5285.348467] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.350411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.354693] Lustre: Skipped 3 previous similar messages [ 5285.359296] Lustre: Skipped 1 previous similar message [ 5290.756299] LustreError: 261458:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5290.760026] LustreError: 261458:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5290.809551] Lustre: server umount lustre-MDT0000 complete [ 5292.723513] LustreError: 259065:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369721 with bad export cookie 9607511815121418058 [ 5292.727720] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5292.730569] LustreError: 259065:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5292.949706] Lustre: server umount lustre-MDT0001 complete [ 5304.790966] Lustre: server umount lustre-OST0000 complete [ 5316.773681] Lustre: server umount lustre-OST0001 complete [ 5324.027978] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5328.687381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.910224] LustreError: 263226:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.952856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.956903] Lustre: Skipped 3 previous similar messages [ 5330.894356] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5335.221829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5337.351132] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5338.769445] Lustre: 264321:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5341.681850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5344.089367] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5344.932511] LustreError: 264677:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.944061] LustreError: 264677:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5348.099559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5350.773565] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5354.607156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5354.609288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5359.969440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5361.628058] Lustre: 266164:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5368.709333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5368.729975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5372.765170] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 11:10:01 (1753369801) [ 5375.458730] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.460447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5375.465573] Lustre: Skipped 3 previous similar messages [ 5375.468392] Lustre: Skipped 7 previous similar messages [ 5380.290873] LustreError: 266848:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5380.293916] LustreError: 266848:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5380.337358] Lustre: server umount lustre-MDT0000 complete [ 5380.581127] LustreError: 263225:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5380.589162] LustreError: 263225:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5382.025229] LustreError: 263205:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369810 with bad export cookie 9607511815121424330 [ 5382.028178] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5382.030567] LustreError: 263205:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5382.215245] Lustre: server umount lustre-MDT0001 complete [ 5393.933378] Lustre: server umount lustre-OST0000 complete [ 5404.930806] Lustre: server umount lustre-OST0001 complete [ 5412.314478] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 5416.525257] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5420.948595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5423.962987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5426.662463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5429.409217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5432.611158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5432.636607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.728794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5432.745185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5432.786648] Lustre: lustre-MDT0000: new disk, initializing [ 5432.831275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5434.483938] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5439.703578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5439.738855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5439.786600] Lustre: 270274:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5439.945944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5441.749458] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5446.030263] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5447.714721] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5450.981215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5451.025311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5453.168052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5454.245925] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5458.191042] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5459.025508] Lustre: Mounted lustre-client [ 5459.393164] LustreError: 272762:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9bd081617800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5459.425208] Lustre: Unmounted lustre-client [ 5463.520411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5463.520843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.526138] LustreError: Skipped 1 previous similar message [ 5463.526951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5463.530642] Lustre: Skipped 2 previous similar messages [ 5463.539112] Lustre: Skipped 3 previous similar messages [ 5469.051241] Lustre: server umount lustre-OST0000 complete [ 5485.023156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5485.108694] Lustre: server umount lustre-MDT0000 complete [ 5486.919833] LustreError: 271049:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369915 with bad export cookie 9607511815121431799 [ 5486.923029] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5486.925448] LustreError: 271049:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5487.094861] Lustre: server umount lustre-MDT0001 complete [ 5493.078263] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5494.337343] Key type lgssc unregistered [ 5512.927751] LNet: 274034:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5588.709598] LNet: Removed LNI 192.168.201.123@tcp [ 5589.077825] Key type .llcrypt unregistered [ 5589.078895] Key type ._llcrypt unregistered [ 5595.939105] Key type ._llcrypt registered [ 5595.940714] Key type .llcrypt registered [ 5595.996269] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 5603.174614] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5603.552219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5603.604437] alg: No test for adler32 (adler32-zlib) [ 5604.490540] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 5604.597599] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5604.600502] LNet: Accept secure, port 988 [ 5606.191262] Key type lgssc registered [ 5606.624496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5610.552731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.293978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5616.012629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5618.628179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5619.370669] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 11:14:07 (1753370047) [ 5640.275123] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 11:14:28 (1753370068) [ 5657.534915] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5658.863241] Key type lgssc unregistered [ 5659.025737] LNet: 279725:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5659.038411] LNet: Removed LNI 192.168.201.123@tcp [ 5659.388660] Key type .llcrypt unregistered [ 5659.390550] Key type ._llcrypt unregistered [ 5692.439958] Key type ._llcrypt registered [ 5692.441591] Key type .llcrypt registered [ 5692.494861] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5692.623567] Key type .llcrypt unregistered [ 5692.624837] Key type ._llcrypt unregistered [ 5706.023179] Key type ._llcrypt registered [ 5706.025850] Key type .llcrypt registered [ 5706.070593] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5706.221252] Key type .llcrypt unregistered [ 5706.222909] Key type ._llcrypt unregistered [ 5739.209828] Key type ._llcrypt registered [ 5739.211512] Key type .llcrypt registered [ 5739.259647] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5739.375302] Key type .llcrypt unregistered [ 5739.376750] Key type ._llcrypt unregistered [ 5752.712356] Key type ._llcrypt registered [ 5752.714172] Key type .llcrypt registered [ 5752.771231] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5752.929593] Key type .llcrypt unregistered [ 5752.931158] Key type ._llcrypt unregistered [ 5766.257098] Key type ._llcrypt registered [ 5766.258767] Key type .llcrypt registered [ 5766.303780] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5766.435324] Key type .llcrypt unregistered [ 5766.437050] Key type ._llcrypt unregistered [ 5797.385765] Key type ._llcrypt registered [ 5797.387619] Key type .llcrypt registered [ 5797.439481] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5797.579601] Key type .llcrypt unregistered [ 5797.581118] Key type ._llcrypt unregistered [ 5810.048911] Key type ._llcrypt registered [ 5810.050459] Key type .llcrypt registered [ 5810.092758] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5810.214544] Key type .llcrypt unregistered [ 5810.216106] Key type ._llcrypt unregistered [ 5821.911162] Key type ._llcrypt registered [ 5821.912610] Key type .llcrypt registered [ 5821.954752] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5822.063398] Key type .llcrypt unregistered [ 5822.064715] Key type ._llcrypt unregistered [ 5842.607468] Key type ._llcrypt registered [ 5842.608963] Key type .llcrypt registered [ 5842.654814] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 5849.243497] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5849.593532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5849.639672] alg: No test for adler32 (adler32-zlib) [ 5850.509587] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 5850.600687] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5850.603105] LNet: Accept secure, port 988 [ 5852.183183] Key type lgssc registered [ 5852.594420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5856.405555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5859.549509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5862.034841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5864.884373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.619206] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5707 sec ========== 11:18:13 (1753370293) [ 5866.310966] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:18:14 (1753370294) === [ 5867.638772] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:18:16 (1753370296) === [ 5883.232664] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5884.542433] Key type lgssc unregistered [ 5884.689422] LNet: 295525:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5884.699477] LNet: Removed LNI 192.168.201.123@tcp [ 5885.060891] Key type .llcrypt unregistered [ 5885.062849] Key type ._llcrypt unregistered