[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 699810506 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003296] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.009469] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012137] LSM: Security Framework initializing [ 0.013265] Yama: becoming mindful. [ 0.015021] SELinux: Initializing. [ 0.016108] *** VALIDATE selinux *** [ 0.029691] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034305] *** VALIDATE tmpfs *** [ 0.036184] *** VALIDATE proc *** [ 0.037265] *** VALIDATE cgroup *** [ 0.038009] *** VALIDATE cgroup2 *** [ 0.039244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042025] Spectre V2 : User space: Vulnerable [ 0.043007] Speculative Store Bypass: Vulnerable [ 0.047007] debug: unmapping init [mem 0xffffffffa6259000-0xffffffffa6260fff] [ 0.050140] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051744] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052017] ... version: 2 [ 0.053012] ... bit width: 48 [ 0.054010] ... generic registers: 4 [ 0.055013] ... value mask: 0000ffffffffffff [ 0.056012] ... max period: 00007fffffffffff [ 0.057012] ... fixed-purpose events: 3 [ 0.058010] ... event mask: 000000070000000f [ 0.059545] rcu: Hierarchical SRCU implementation. [ 0.061743] smp: Bringing up secondary CPUs ... [ 0.062623] x86: Booting SMP configuration: [ 0.063030] .... node #0, CPUs: #1 #2 #3 [ 0.067576] smp: Brought up 1 node, 4 CPUs [ 0.069015] smpboot: Max logical packages: 1 [ 0.070216] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.160036] node 0 deferred pages initialised in 86ms [ 0.164685] devtmpfs: initialized [ 0.169233] x86/mm: Memory block size: 128MB [ 0.172825] gcov: version magic: 0x41383552 [ 0.176310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.185389] pinctrl core: initialized pinctrl subsystem [ 0.188222] [ 0.188882] ************************************************************* [ 0.192013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194010] ** ** [ 0.198016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201013] ** ** [ 0.204015] ** This means that this kernel is built to expose internal ** [ 0.207012] ** IOMMU data structures, which may compromise security on ** [ 0.211013] ** your system. ** [ 0.213011] ** ** [ 0.216012] ** If you see this message and you are not debugging the ** [ 0.219013] ** kernel, report this immediately to your vendor! ** [ 0.222012] ** ** [ 0.224009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227011] ************************************************************* [ 0.229718] NET: Registered protocol family 16 [ 0.232442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.236052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244068] cpuidle: using governor menu [ 0.246000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.249507] PCI: Using configuration type 1 for base access [ 0.252149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265454] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.269045] cryptd: max_cpu_qlen set to 1000 [ 0.275643] ACPI: Added _OSI(Module Device) [ 0.277013] ACPI: Added _OSI(Processor Device) [ 0.280011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289029] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297529] ACPI: Interpreter enabled [ 0.299057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.300008] ACPI: Using IOAPIC for interrupt routing [ 0.301091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.303388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.309237] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.312034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.315014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.319074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.325439] acpiphp: Slot [2] registered [ 0.327102] acpiphp: Slot [5] registered [ 0.329123] acpiphp: Slot [6] registered [ 0.331098] acpiphp: Slot [7] registered [ 0.333108] acpiphp: Slot [8] registered [ 0.335118] acpiphp: Slot [9] registered [ 0.337377] acpiphp: Slot [10] registered [ 0.340169] acpiphp: Slot [3] registered [ 0.343108] acpiphp: Slot [4] registered [ 0.345091] acpiphp: Slot [11] registered [ 0.347118] acpiphp: Slot [12] registered [ 0.349083] acpiphp: Slot [13] registered [ 0.351085] acpiphp: Slot [14] registered [ 0.353335] acpiphp: Slot [15] registered [ 0.355151] acpiphp: Slot [16] registered [ 0.357081] acpiphp: Slot [17] registered [ 0.359083] acpiphp: Slot [18] registered [ 0.361082] acpiphp: Slot [19] registered [ 0.363083] acpiphp: Slot [20] registered [ 0.365304] acpiphp: Slot [21] registered [ 0.367263] acpiphp: Slot [22] registered [ 0.369094] acpiphp: Slot [23] registered [ 0.371079] acpiphp: Slot [24] registered [ 0.373082] acpiphp: Slot [25] registered [ 0.376087] acpiphp: Slot [26] registered [ 0.377080] acpiphp: Slot [27] registered [ 0.379111] acpiphp: Slot [28] registered [ 0.381084] acpiphp: Slot [29] registered [ 0.383057] acpiphp: Slot [30] registered [ 0.385096] acpiphp: Slot [31] registered [ 0.386084] PCI host bridge to bus 0000:00 [ 0.387025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.395029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.400031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.404021] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.409034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.413223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.417955] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.422251] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.436017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.443079] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.445025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.449039] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.451029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.455935] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459456] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.463035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.467078] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.492015] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.501023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508976] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.523017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.539021] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.553294] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.562014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.570014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.589017] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.599000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.605014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.633017] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.651528] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.663015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.674022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.699019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.716066] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.730053] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.744057] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.764016] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.780799] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.791015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.816023] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.829295] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.832709] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.839398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.842296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.850375] iommu: Default domain type: Passthrough [ 0.855444] SCSI subsystem initialized [ 0.860286] ACPI: bus type USB registered [ 0.863394] usbcore: registered new interface driver usbfs [ 0.868078] usbcore: registered new interface driver hub [ 0.869000] usbcore: registered new device driver usb [ 0.870435] pps_core: LinuxPPS API ver. 1 registered [ 0.873009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.876077] PTP clock support registered [ 0.879096] EDAC MC: Ver: 3.0.0 [ 0.882799] PCI: Using ACPI for IRQ routing [ 0.885000] NetLabel: Initializing [ 0.885000] NetLabel: domain hash size = 128 [ 0.885000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.886080] NetLabel: unlabeled traffic allowed by default [ 0.888270] vgaarb: loaded [ 0.891145] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.893010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.905099] clocksource: Switched to clocksource kvm-clock [ 1.052879] VFS: Disk quotas dquot_6.6.0 [ 1.055823] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.061213] *** VALIDATE ramfs *** [ 1.062565] *** VALIDATE hugetlbfs *** [ 1.064220] pnp: PnP ACPI init [ 1.066659] pnp: PnP ACPI: found 6 devices [ 1.086691] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.090332] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.092571] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.095460] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.098041] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.101067] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.105780] NET: Registered protocol family 2 [ 1.107914] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.114356] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.118548] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.125712] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.129919] TCP: Hash tables configured (established 65536 bind 65536) [ 1.133545] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.137412] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.140403] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.144804] NET: Registered protocol family 1 [ 1.149595] RPC: Registered named UNIX socket transport module. [ 1.153575] RPC: Registered udp transport module. [ 1.155511] RPC: Registered tcp transport module. [ 1.158136] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.161165] NET: Registered protocol family 44 [ 1.167278] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.172368] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.177424] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.184574] PCI: CLS 0 bytes, default 64 [ 1.188497] Unpacking initramfs... [ 3.167863] debug: unmapping init [mem 0xffff9c78fcc54000-0xffff9c78fffbffff] [ 3.172696] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.175351] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.179434] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.846429] Initialise system trusted keyrings [ 3.848677] Key type blacklist registered [ 3.851048] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.863372] zbud: loaded [ 3.867033] *** VALIDATE nfs *** [ 3.868531] *** VALIDATE nfs4 *** [ 3.871035] pstore: using deflate compression [ 3.891293] Platform Keyring initialized [ 4.099583] NET: Registered protocol family 38 [ 4.102339] Key type asymmetric registered [ 4.105728] Asymmetric key parser 'x509' registered [ 4.111378] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.118384] io scheduler mq-deadline registered [ 4.122770] io scheduler kyber registered [ 4.126977] io scheduler bfq registered [ 4.131784] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.141782] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.146131] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.149539] ACPI: Power Button [PWRF] [ 4.311540] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.433544] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.665717] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.816600] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.105437] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.145632] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.190993] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.197693] Non-volatile memory driver v1.3 [ 5.199881] Linux agpgart interface v0.103 [ 5.241419] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 5.245309] vda: detected capacity change from 0 to 68505600 [ 5.264157] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.267560] vdb: detected capacity change from 0 to 1073741824 [ 5.296392] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.301519] vdc: detected capacity change from 0 to 2621440000 [ 5.319956] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.323621] vdd: detected capacity change from 0 to 2621440000 [ 5.344882] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.348843] vde: detected capacity change from 0 to 4294967296 [ 5.373833] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.378920] vdf: detected capacity change from 0 to 4294967296 [ 5.389574] libphy: Fixed MDIO Bus: probed [ 5.400423] usbcore: registered new interface driver usbserial_generic [ 5.403825] usbserial: USB Serial support registered for generic [ 5.408788] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.418924] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.420797] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.425495] mousedev: PS/2 mouse device common for all mice [ 5.431124] rtc_cmos 00:05: RTC can wake from S4 [ 5.434307] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.436512] rtc_cmos 00:05: registered as rtc0 [ 5.443676] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.443821] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.451035] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.453415] intel_pstate: CPU model not supported [ 5.462291] hid: raw HID events driver (C) Jiri Kosina [ 5.467164] usbcore: registered new interface driver usbhid [ 5.470157] usbhid: USB HID core driver [ 5.472560] drop_monitor: Initializing network drop monitor service [ 5.475904] Initializing XFRM netlink socket [ 5.478681] NET: Registered protocol family 10 [ 5.483324] Segment Routing with IPv6 [ 5.485484] NET: Registered protocol family 17 [ 5.493426] mpls_gso: MPLS GSO support [ 5.502317] RAS: Correctable Errors collector initialized. [ 5.504616] AVX version of gcm_enc/dec engaged. [ 5.506332] AES CTR mode by8 optimization enabled [ 5.648407] sched_clock: Marking stable (5648384439, 0)->(7108931889, -1460547450) [ 5.654458] registered taskstats version 1 [ 5.656929] Loading compiled-in X.509 certificates [ 5.661215] zswap: loaded using pool lzo/zbud [ 5.693333] Key type big_key registered [ 5.720705] Key type encrypted registered [ 5.723121] ima: No TPM chip found, activating TPM-bypass! [ 5.728533] ima: Allocated hash algorithm: sha1 [ 5.733504] ima: No architecture policies found [ 5.737399] evm: Initialising EVM extended attributes: [ 5.741542] evm: security.selinux [ 5.744459] evm: security.ima [ 5.747416] evm: security.capability [ 5.750758] evm: HMAC attrs: 0x1 [ 5.755650] rtc_cmos 00:05: setting system clock to 2025-10-07 04:55:22 UTC (1759812922) [ 5.763722] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 5.769547] debug: unmapping init [mem 0xffffffffa5f82000-0xffffffffa6258fff] [ 5.780163] Write protecting the kernel read-only data: 28672k [ 5.786697] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 5.792300] debug: unmapping init [mem 0xffffffffa4f14000-0xffffffffa4ffffff] [ 5.850840] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.864514] systemd[1]: Detected virtualization kvm. [ 5.867780] systemd[1]: Detected architecture x86-64. [ 5.871763] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.907710] systemd[1]: No hostname configured. [ 5.910565] systemd[1]: Set hostname to . [ 5.913962] random: systemd: uninitialized urandom read (16 bytes read) [ 5.917570] systemd[1]: Initializing machine ID from random generator. [ 5.996956] random: ln: uninitialized urandom read (6 bytes read) [ 6.187930] random: systemd: uninitialized urandom read (16 bytes read) [ 6.192454] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.198723] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.209170] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.458334] device-mapper: uevent: version 1.0.3 [ 7.460837] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 7.996470] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.758608] virtio_net virtio0 ens2: renamed from eth0 [ 9.153709] scsi host0: ata_piix [ 9.304320] scsi host1: ata_piix [ 9.307345] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.312649] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.267897] random: crng init done [ 14.269367] random: 7 urandom warning(s) missed due to ratelimiting [ 15.002357] dracut-initqueue[602]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.876500] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.641656] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.995774] SELinux: Disabled at runtime. [ 18.078747] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.090645] systemd[1]: Detected virtualization kvm. [ 18.092689] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.922587] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.932547] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.944819] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.953575] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.962507] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.980224] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.996743] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ 19.348322] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.093435] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.779137] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.797655] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.341096] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.462110] EDAC sbridge: Ver: 1.1.2 [ 24.510203] Key type dns_resolver registered [ 25.004218] NFS: Registering the id_resolver key type [ 25.006626] Key type id_resolver registered [ 25.008294] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-server login: [ 114.999686] libcfs: loading out-of-tree module taints kernel. [ 115.026342] Key type ._llcrypt registered [ 115.029248] Key type .llcrypt registered [ 115.116827] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 130.346482] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 131.559635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.582438] alg: No test for adler32 (adler32-zlib) [ 132.854443] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 133.476619] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 135.167179] Key type lgssc registered [ 136.778130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.423664] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.881547] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.062612] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.831239] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.134203] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.310195] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 207.122102] hrtimer: interrupt took 2911951 ns [ 212.191440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.276096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.351133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.000591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.120924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.509953] Lustre: lustre-MDT0000: new disk, initializing [ 214.781198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.843742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.476708] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.200202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 245.331166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 245.497730] Lustre: 6532:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 245.585958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 245.601339] Lustre: Skipped 1 previous similar message [ 245.831528] Lustre: lustre-MDT0001: new disk, initializing [ 245.957394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.013069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 246.018572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 250.952664] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.644957] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 268.227883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 268.370566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 268.749157] Lustre: lustre-OST0000: new disk, initializing [ 268.781875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.877637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.884276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 269.901967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.977331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 275.359655] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.963704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 292.069279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 292.253864] Lustre: lustre-OST0001: new disk, initializing [ 292.263379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 292.355932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.541313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 298.554885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 298.648119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 299.757267] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.072293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 320.725657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 332.217430] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing check_logdir /tmp/testlogs/ [ 337.786901] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing yml_node [ 342.447905] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 345.563782] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 348.292317] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 350.165723] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 7 01:01:05 EDT 2025 [ 369.715674] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 371.481671] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 389.606620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 389.625101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.669886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.624472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.624941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.743905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 395.759882] Lustre: Skipped 6 previous similar messages [ 400.868356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.893056] Lustre: Skipped 3 previous similar messages [ 402.399143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 402.541215] LustreError: 12256:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 402.701107] Lustre: server umount lustre-MDT0000 complete [ 406.966459] LustreError: 6525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813323 with bad export cookie 3272914289825826063 [ 406.968227] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.981872] LustreError: 6525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 411.107204] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.111285] LustreError: 6540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.137421] Lustre: Skipped 2 previous similar messages [ 411.138571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 411.145646] LustreError: 6540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 411.156072] Lustre: Skipped 1 previous similar message [ 416.224400] LustreError: 6539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.260194] LustreError: 6539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 417.253350] LustreError: 12258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 417.275532] LustreError: 12258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 421.344131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.484793] LustreError: 12460:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 421.494536] LustreError: 12460:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 421.654475] Lustre: server umount lustre-MDT0001 complete [ 426.298251] LustreError: 12663:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 426.317724] LustreError: 12663:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 426.518266] Lustre: server umount lustre-OST0000 complete [ 430.813655] LustreError: 12864:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 430.817733] LustreError: 12864:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 431.158150] Lustre: server umount lustre-OST0001 complete [ 438.784896] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 450.880073] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 463.136861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 470.695317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 477.785631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 485.002882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 493.671652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 493.794581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.153509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 494.194556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 494.271636] Lustre: lustre-MDT0000: new disk, initializing [ 494.416237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.454554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 499.335946] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 511.394634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 511.475531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 511.579882] Lustre: 15050:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 511.648911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 511.653071] Lustre: Skipped 1 previous similar message [ 511.735804] Lustre: lustre-MDT0001: new disk, initializing [ 511.908538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 511.955487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 511.982205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 517.604826] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 526.643764] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 531.561901] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 538.881154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 539.019792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 539.429957] Lustre: lustre-OST0000: new disk, initializing [ 539.439293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 539.550199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.402829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 541.425294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 541.624730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 546.159359] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 553.835267] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 558.190927] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 558.482233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 563.013081] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 563.302086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 567.265851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.272669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.281789] Lustre: Skipped 1 previous similar message [ 567.287973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.290736] Lustre: Skipped 4 previous similar messages [ 571.399454] LustreError: 17991:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 571.413801] LustreError: 17991:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 571.636558] Lustre: server umount lustre-OST0000 complete [ 577.512399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.532852] Lustre: Skipped 2 previous similar messages [ 581.967723] LustreError: 18194:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 581.972402] LustreError: 18194:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 582.056775] Lustre: server umount lustre-MDT0000 complete [ 582.628468] LustreError: 15845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.652854] LustreError: 15845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 586.393663] LustreError: 15042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813503 with bad export cookie 3272914289825827589 [ 586.398888] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.403706] LustreError: 15042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 586.912331] Lustre: server umount lustre-MDT0001 complete [ 594.068757] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:05:08 (1759813508) [ 614.434717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.080651] LustreError: 19531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.224644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.133962] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 624.613532] LustreError: 19530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.626820] LustreError: 19530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 628.117077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 632.470653] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 638.673838] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 643.928516] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 646.609336] Lustre: Failing over lustre-MDT0000 [ 646.634255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.666930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.671277] Lustre: Skipped 2 previous similar messages [ 646.777271] LustreError: 20876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 646.782855] LustreError: 20876:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 647.028245] Lustre: server umount lustre-MDT0000 complete [ 651.747778] LustreError: 20554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.776281] LustreError: 20554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 668.127347] Lustre: 3674:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759813568/real 1759813568] req@ffff9c797bcfe300 x1845297728888320/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759813584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 668.176882] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.993701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.344202] LustreError: 3670:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c7842607800 x1845297728891008/t0(0) o250->MGC192.168.201.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.738852] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 681.780342] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 682.990987] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 683.000667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 683.045945] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 689.484675] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 698.337436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.337928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 698.340321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 698.340330] Lustre: Skipped 1 previous similar message [ 698.349345] Lustre: Skipped 2 previous similar messages [ 698.368875] LustreError: Skipped 1 previous similar message [ 700.423455] LustreError: 21861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 700.426546] LustreError: 21861:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 700.518266] Lustre: server umount lustre-MDT0000 complete [ 704.371144] LustreError: 21862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813621 with bad export cookie 3272914289825829010 [ 704.373228] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.410067] LustreError: 21862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 704.887282] Lustre: server umount lustre-MDT0001 complete [ 713.214090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.544256] LustreError: 22540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.562949] LustreError: 22540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 713.613549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.617246] Lustre: Skipped 2 previous similar messages [ 717.744309] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 726.622700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 730.706801] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 734.962108] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 738.061094] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 739.786179] Lustre: Failing over lustre-MDT0000 [ 739.988689] Lustre: server umount lustre-MDT0000 complete [ 742.368798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 742.375505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.652827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.760741] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.955393] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 759.657356] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 762.340127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 762.342211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 762.347720] Lustre: Skipped 1 previous similar message [ 762.355306] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 764.894300] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 771.555047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.560482] Lustre: Skipped 4 previous similar messages [ 782.311550] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 782.397047] LustreError: 24863:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 782.401249] LustreError: 24863:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 782.457099] Lustre: server umount lustre-MDT0000 complete [ 782.819045] LustreError: 22540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.849992] LustreError: 22540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 785.712126] LustreError: 22521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813702 with bad export cookie 3272914289825829941 [ 785.720380] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.729207] LustreError: 22521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.035583] Lustre: server umount lustre-MDT0001 complete [ 793.800874] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:08:28 (1759813708) [ 805.763290] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 812.745887] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 818.702750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 823.080852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 827.578796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 831.889604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 838.495926] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 845.094038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 845.147847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.308312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 845.332967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 845.386277] Lustre: lustre-MDT0000: new disk, initializing [ 845.456856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.461789] Lustre: Skipped 2 previous similar messages [ 845.474739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 847.779268] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 854.544607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.590237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 854.654955] Lustre: 28532:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 854.829164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 857.328213] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 861.730809] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 863.636086] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 867.613428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 867.653699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 869.043664] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 869.097378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 869.107769] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 871.056206] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 876.055363] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 881.215664] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 892.997941] Lustre: server umount lustre-OST0000 complete [ 895.968624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 895.973217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.978877] Lustre: Skipped 3 previous similar messages [ 900.065730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.068405] Lustre: Skipped 5 previous similar messages [ 905.253607] Lustre: server umount lustre-MDT0000 complete [ 907.514939] LustreError: 29313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813824 with bad export cookie 3272914289825830333 [ 907.521868] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.527658] LustreError: 29313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 917.994201] LustreError: 31575:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 917.999145] LustreError: 31575:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 918.162038] Lustre: server umount lustre-MDT0001 complete [ 924.521458] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 926.161890] Key type lgssc unregistered [ 926.373722] LNet: 32109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.379417] LNetError: 32109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.392533] LNet: Removed LNI 192.168.201.121@tcp [ 926.955148] Key type .llcrypt unregistered [ 926.956739] Key type ._llcrypt unregistered [ 942.090952] Key type ._llcrypt registered [ 942.093167] Key type .llcrypt registered [ 942.173962] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 942.835082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.848971] alg: No test for adler32 (adler32-zlib) [ 943.777740] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 943.910875] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 945.535209] Key type lgssc registered [ 946.225405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 950.657411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 950.667974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.941429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.994190] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 958.463308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 958.675184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 960.772412] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 963.702328] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 965.854190] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 969.639295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 969.882528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.682395] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 975.652903] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 984.236366] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 992.982489] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:11:48 (1759813908) [ 994.025367] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 995.092139] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 998.873450] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:11:54 (1759813914) [ 1003.593891] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:11:59 (1759813919) [ 1004.166489] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1009.082804] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1009.086991] Lustre: Skipped 2 previous similar messages [ 1010.160734] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:12:05 (1759813925) [ 1014.704743] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1014.710314] Lustre: Skipped 4 previous similar messages [ 1023.435022] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:12:19 (1759813939) [ 1025.061959] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1026.981173] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1031.888332] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:12:27 (1759813947) [ 1032.847896] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1032.850071] Lustre: Skipped 9 previous similar messages [ 1039.561851] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:12:35 (1759813955) [ 1049.058212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.066029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.076202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.105571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1051.107589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.112055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.399169] LustreError: 40048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1052.487000] Lustre: server umount lustre-OST0000 complete [ 1056.224929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.226204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.226495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.226499] Lustre: Skipped 1 previous similar message [ 1056.229801] Lustre: Skipped 1 previous similar message [ 1060.561904] LustreError: 40251:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1060.564952] LustreError: 40251:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1060.624068] Lustre: server umount lustre-MDT0000 complete [ 1061.345662] LustreError: 34194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.359976] LustreError: 34194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1062.561537] LustreError: 33459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813979 with bad export cookie 17261352968406588906 [ 1062.563837] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.569863] LustreError: 33459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1062.658606] LustreError: 40452:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1062.662162] LustreError: 40452:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1062.781057] Lustre: server umount lustre-MDT0001 complete [ 1068.177160] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1069.665449] Key type lgssc unregistered [ 1069.852049] LNet: 40983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.856673] LNetError: 40983:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.865672] LNet: Removed LNI 192.168.201.121@tcp [ 1070.303463] Key type .llcrypt unregistered [ 1070.305170] Key type ._llcrypt unregistered [ 1079.246556] Key type ._llcrypt registered [ 1079.248732] Key type .llcrypt registered [ 1079.309517] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 1088.740980] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1089.180473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1089.212183] alg: No test for adler32 (adler32-zlib) [ 1090.157208] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1090.315319] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1091.935201] Key type lgssc registered [ 1092.576284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1097.740921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1101.140882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1104.186182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1107.507023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1111.437672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1111.474202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1111.484596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.629452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1112.646069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1112.697176] Lustre: lustre-MDT0000: new disk, initializing [ 1112.754086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.765635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1114.641171] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1120.379045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1120.431025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1120.490792] Lustre: 44086:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1120.527764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1120.530901] Lustre: Skipped 1 previous similar message [ 1120.584242] Lustre: lustre-MDT0001: new disk, initializing [ 1120.637768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.655473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1120.661114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1122.590153] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1126.737659] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1128.667380] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1131.930159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1131.981572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1132.111795] Lustre: lustre-OST0000: new disk, initializing [ 1132.115027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1132.159291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.899682] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1135.837292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1135.843477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1135.898440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1139.076595] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.102381] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1141.226496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1143.243468] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1143.358652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1146.337104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1146.339149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.344799] LustreError: Skipped 1 previous similar message [ 1146.351909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.403705] LustreError: 47024:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1150.516909] Lustre: server umount lustre-OST0000 complete [ 1156.577156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.577356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.583839] Lustre: Skipped 2 previous similar messages [ 1156.584099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.584103] Lustre: Skipped 1 previous similar message [ 1158.878137] LustreError: 47225:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1158.882281] LustreError: 47225:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1158.940479] Lustre: server umount lustre-MDT0000 complete [ 1160.777151] LustreError: 44077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814077 with bad export cookie 18302963123983539346 [ 1160.780275] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.781977] LustreError: 44077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1160.859496] LustreError: 47427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1160.863847] LustreError: 47427:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1160.992968] Lustre: server umount lustre-MDT0001 complete [ 1166.131151] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:14:41 (1759814081) [ 1169.711639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.986264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.943580] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1175.011808] LustreError: 48146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.020507] LustreError: 48146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1176.285893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.487885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1178.442248] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1181.239894] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1183.152402] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1186.680555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1186.895693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.674193] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1192.325682] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1204.643343] LustreError: 50365:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 1205.078084] Lustre: Modifying parameter general.timeout=1 in log params [ 1206.302287] Lustre: Modifying parameter general.timeout=4 in log params [ 1206.305237] Lustre: Skipped 2 previous similar messages [ 1208.330904] Lustre: Modifying parameter general.timeout=9 in log params [ 1208.333135] Lustre: Skipped 4 previous similar messages [ 1212.614863] Lustre: Modifying parameter general.timeout=19 in log params [ 1212.617740] Lustre: Skipped 9 previous similar messages [ 1220.824235] Lustre: Modifying parameter general.timeout=39 in log params [ 1220.827685] Lustre: Skipped 19 previous similar messages [ 1236.996593] Lustre: Modifying parameter general.timeout=79 in log params [ 1236.999517] Lustre: Skipped 39 previous similar messages [ 1262.098295] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:16:17 (1759814177) [ 1263.364422] Lustre: 57271:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1264.971681] Lustre: 57467:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1264.977596] Lustre: 57467:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 1267.486029] Lustre: 57782:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1267.491530] Lustre: 57782:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 2 previous similar messages [ 1269.290950] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1269.295968] Lustre: Skipped 59 previous similar messages [ 1280.711565] Lustre: 59121:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1280.722526] LustreError: 59121:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1282.730649] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:16:38 (1759814198) [ 1289.183840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1289.187909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.195398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.199113] Lustre: Skipped 1 previous similar message [ 1294.022450] LustreError: 59819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1294.027328] LustreError: 59819:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1294.077247] Lustre: server umount lustre-MDT0000 complete [ 1295.778307] LustreError: 53319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814212 with bad export cookie 18302963123983540228 [ 1295.780744] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.784915] LustreError: 53319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1295.840150] LustreError: 48141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.840540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.854151] Lustre: Skipped 2 previous similar messages [ 1295.856611] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1295.860285] Lustre: Skipped 3 previous similar messages [ 1295.970039] Lustre: server umount lustre-MDT0001 complete [ 1303.791536] LustreError: 60224:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1303.795115] LustreError: 60224:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1303.881162] Lustre: server umount lustre-OST0000 complete [ 1307.849185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1309.624995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.379960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.104982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1316.418850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.558363] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1316.697909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.508275] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1322.600507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1322.665306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1322.686753] Lustre: 61671:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1324.786604] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1327.335335] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.103599] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1332.201856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1332.310495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1335.061554] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1337.609635] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1343.738659] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1343.743116] Lustre: Skipped 14 previous similar messages [ 1343.760137] Lustre: 63907:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1346.528709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1346.530746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.532448] LustreError: Skipped 1 previous similar message [ 1346.533237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.539616] Lustre: Skipped 1 previous similar message [ 1348.206099] LustreError: 64152:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1348.209091] LustreError: 64152:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1348.283257] Lustre: server umount lustre-OST0000 complete [ 1351.975200] Lustre: server umount lustre-MDT0000 complete [ 1353.625971] LustreError: 61662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814270 with bad export cookie 18302963123983542650 [ 1353.627570] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.632937] LustreError: 61662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1353.807281] Lustre: server umount lustre-MDT0001 complete [ 1358.546511] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1359.852728] Key type lgssc unregistered [ 1360.012387] LNet: 65089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.015880] LNetError: 65089:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.024952] LNet: Removed LNI 192.168.201.121@tcp [ 1360.406647] Key type .llcrypt unregistered [ 1360.407873] Key type ._llcrypt unregistered [ 1372.420516] Key type ._llcrypt registered [ 1372.422307] Key type .llcrypt registered [ 1372.472763] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1372.931501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.964544] alg: No test for adler32 (adler32-zlib) [ 1373.858590] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1373.972373] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1375.567147] Key type lgssc registered [ 1376.063334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1379.650169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1379.660527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.912791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.860661] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1386.896929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1387.164957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.057358] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1391.451538] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1393.068154] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1396.077894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1396.325637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.828326] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1401.413939] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.788967] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1417.190550] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.192774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.201878] Lustre: Skipped 1 previous similar message [ 1420.300823] LustreError: 69168:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1420.358606] Lustre: server umount lustre-MDT0000 complete [ 1422.157358] LustreError: 68813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814338 with bad export cookie 1419995376311046429 [ 1422.159881] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.162308] LustreError: 68813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1422.240462] LustreError: 69371:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1422.244461] LustreError: 69371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1422.349281] Lustre: server umount lustre-MDT0001 complete [ 1433.331766] LustreError: 69571:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1433.334422] LustreError: 69571:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1433.416709] Lustre: server umount lustre-OST0000 complete [ 1437.278274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1438.999870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1440.689452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1442.340653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1445.391590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.517224] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1445.532947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1445.682194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.453398] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1451.397763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1451.450075] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1451.468531] Lustre: 71019:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1451.724265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.554463] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1455.934633] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1457.385166] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1460.246579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1460.338949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1460.518792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.244368] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1465.595179] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1466.313200] Lustre: 73263:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1467.150297] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1473.505556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1473.505840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.510125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.514903] Lustre: Skipped 2 previous similar messages [ 1473.517636] Lustre: Skipped 2 previous similar messages [ 1478.626854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.630736] Lustre: Skipped 1 previous similar message [ 1483.231151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1483.250082] LustreError: 73511:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1483.253239] LustreError: 73511:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1483.326838] Lustre: server umount lustre-OST0000 complete [ 1487.328166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1487.331951] LustreError: Skipped 1 previous similar message [ 1487.333762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.339230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.009717] LustreError: 73721:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1491.012300] LustreError: 73721:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1491.040479] Lustre: server umount lustre-MDT0000 complete [ 1492.528436] LustreError: 71752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814409 with bad export cookie 1419995376311047241 [ 1492.531251] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.532955] LustreError: 71752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1492.699161] Lustre: server umount lustre-MDT0001 complete [ 1497.284858] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1498.608905] Key type lgssc unregistered [ 1498.762653] LNet: 74453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.766359] LNetError: 74453:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.774382] LNet: Removed LNI 192.168.201.121@tcp [ 1499.150630] Key type .llcrypt unregistered [ 1499.152122] Key type ._llcrypt unregistered [ 1510.129636] Key type ._llcrypt registered [ 1510.131908] Key type .llcrypt registered [ 1510.174146] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1510.532728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1510.541677] alg: No test for adler32 (adler32-zlib) [ 1511.425227] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1511.529372] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1513.119180] Key type lgssc registered [ 1513.616025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1516.905074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1516.913169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.128281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.850776] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1523.605475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1523.910210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1525.638489] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1528.094461] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1529.843178] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1532.828564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1533.085794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.547423] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1537.916688] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1545.846818] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1546.673907] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1546.677277] Lustre: Skipped 1 previous similar message [ 1547.987544] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1547.990955] Lustre: Skipped 2 previous similar messages [ 1550.005084] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1550.010881] Lustre: Skipped 4 previous similar messages [ 1554.351743] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1554.355932] Lustre: Skipped 10 previous similar messages [ 1562.508380] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1562.511438] Lustre: Skipped 18 previous similar messages [ 1578.900422] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1578.902592] Lustre: Skipped 38 previous similar messages [ 1591.162870] Lustre: 83576:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1592.292969] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:21:48 (1759814508) [ 1593.020481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1593.788906] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:21:49 (1759814509) [ 1595.871878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1595.874709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.882078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.947921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.948272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.949148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.961497] Lustre: Skipped 1 previous similar message [ 1601.198184] LustreError: 83872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1601.277345] Lustre: server umount lustre-OST0000 complete [ 1604.064976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.065563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.071339] Lustre: Skipped 1 previous similar message [ 1609.184899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.190079] Lustre: Skipped 2 previous similar messages [ 1614.304254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.306669] Lustre: Skipped 1 previous similar message [ 1617.375128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1617.416635] LustreError: 84073:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1617.419881] LustreError: 84073:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1617.453658] Lustre: server umount lustre-MDT0000 complete [ 1618.936330] LustreError: 75819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814535 with bad export cookie 9313346884805463944 [ 1618.940049] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.940770] LustreError: 75819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1619.017328] LustreError: 84276:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1619.020970] LustreError: 84276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1619.116983] Lustre: server umount lustre-MDT0001 complete [ 1623.717794] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1624.995785] Key type lgssc unregistered [ 1625.144314] LNet: 84807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.147138] LNetError: 84807:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.154280] LNet: Removed LNI 192.168.201.121@tcp [ 1625.538615] Key type .llcrypt unregistered [ 1625.540766] Key type ._llcrypt unregistered [ 1627.220792] Key type ._llcrypt registered [ 1627.222288] Key type .llcrypt registered [ 1627.276775] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1629.039777] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules [ 1629.491300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1629.520260] alg: No test for adler32 (adler32-zlib) [ 1630.400221] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1630.517919] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1632.111173] Key type lgssc registered [ 1632.639570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1633.116048] LustreError: 85924:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1642.224309] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1645.925240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1654.301389] LustreError: 85924:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1654.313941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.552297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.213145] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1660.746373] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:22:56 (1759814576) [ 1662.742368] LustreError: 86962:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1662.747264] LustreError: 86962:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1662.750785] LustreError: 86962:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1662.823555] LustreError: 87734:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1662.920771] Lustre: server umount lustre-MDT0000 complete [ 1669.028869] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1670.435172] Key type lgssc unregistered [ 1670.606562] LNet: 88419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.612373] LNetError: 88419:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.622818] LNet: Removed LNI 192.168.201.121@tcp [ 1671.059628] Key type .llcrypt unregistered [ 1671.061950] Key type ._llcrypt unregistered [ 1679.898151] Key type ._llcrypt registered [ 1679.899378] Key type .llcrypt registered [ 1679.945726] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1680.370936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.437920] alg: No test for adler32 (adler32-zlib) [ 1681.317807] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1681.417906] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1683.007167] Key type lgssc registered [ 1683.459930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1686.952108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1686.964078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.166458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.792983] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1693.432708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1693.742714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1695.502536] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1697.806500] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1699.392611] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1702.179325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1702.454986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.862407] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1707.334379] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1737.299081] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:24:13 (1759814653) [ 1738.108855] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1738.885385] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:24:14 (1759814654) [ 1743.329453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.330292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.333742] Lustre: Skipped 2 previous similar messages [ 1743.335438] Lustre: Skipped 1 previous similar message [ 1746.307287] LustreError: 92504:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1746.342959] Lustre: server umount lustre-MDT0000 complete [ 1747.797415] LustreError: 89574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814664 with bad export cookie 14414773951690561998 [ 1747.801277] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.803020] LustreError: 89574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1747.890719] LustreError: 92705:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1747.894346] LustreError: 92705:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1747.980454] Lustre: server umount lustre-MDT0001 complete [ 1759.731208] LustreError: 92906:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1759.733531] LustreError: 92906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1759.901524] Lustre: server umount lustre-OST0000 complete [ 1764.997252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.186390] LustreError: 93535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1765.195306] LustreError: 93535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1765.268897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.960808] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1770.360430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.618302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1772.268366] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1774.451753] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1775.978034] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1778.571609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1781.484268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1781.514071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1781.604496] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1781.609290] LustreError: 93523:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1781.623510] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1781.628369] LustreError: 95395:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1781.631621] LustreError: 95395:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1781.635701] LustreError: 95395:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1781.707817] Lustre: server umount lustre-OST0000 complete [ 1781.709657] LustreError: 95395:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1785.538307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1785.634307] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1785.639395] LustreError: 93524:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1785.652498] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1785.657799] LustreError: 95861:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1785.661467] LustreError: 95861:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1785.664516] LustreError: 95861:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1785.726683] Lustre: server umount lustre-OST0000 complete [ 1785.729835] LustreError: 95861:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1787.008495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1789.966185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1790.051362] LustreError: 96384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.073697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1790.076140] Lustre: Found index 0 for lustre-OST0000, updating log [ 1790.079400] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1790.109175] Lustre: lustre-OST0000: new disk, initializing [ 1790.111534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1790.275910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.869046] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1794.511193] Lustre: Failing over lustre-OST0000 [ 1794.538178] LustreError: 96944:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1794.542446] LustreError: 96944:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1794.626164] Lustre: server umount lustre-OST0000 complete [ 1800.672682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.673235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.678326] Lustre: Skipped 1 previous similar message [ 1800.681030] Lustre: Skipped 1 previous similar message [ 1805.794717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.798499] Lustre: Skipped 1 previous similar message [ 1805.799096] LustreError: 97144:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1805.803695] LustreError: 97144:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1805.844235] Lustre: server umount lustre-MDT0000 complete [ 1807.660771] LustreError: 93515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814724 with bad export cookie 14414773951690564728 [ 1807.663931] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.665289] LustreError: 93515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1807.836949] Lustre: server umount lustre-MDT0001 complete [ 1812.105527] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:25:27 (1759814727) [ 1819.555666] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1823.842307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.007302] LustreError: 98713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1824.045898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.714798] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1829.348369] LustreError: 98713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.417251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1831.538512] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1832.901111] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1836.070459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.449757] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1839.195602] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1839.523505] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1839.528484] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1839.537271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1843.033155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.057650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.086141] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1843.124799] Lustre: lustre-OST0001: new disk, initializing [ 1843.126705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1843.243509] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 1843.250242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1845.806027] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1846.728157] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 1846.733562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1846.736930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1846.741243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1846.797511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1848.805679] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 1848.811250] Lustre: Skipped 1 previous similar message [ 1848.812517] Lustre: lustre-OST0000: Recovery over after 0:09, of 1 clients 1 recovered and 0 were evicted. [ 1848.812620] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1848.813527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1848.815413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1849.813267] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1857.094396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1864.164178] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.166444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1864.170171] Lustre: Skipped 2 previous similar messages [ 1864.173642] Lustre: Skipped 1 previous similar message [ 1869.281823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1869.285483] Lustre: Skipped 1 previous similar message [ 1870.021368] LustreError: 101956:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1870.024050] LustreError: 101956:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1870.055245] Lustre: server umount lustre-MDT0001 complete [ 1871.194456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1872.654804] LustreError: 98707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1873.998866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1874.043490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1874.046184] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1874.067984] Lustre: 98702:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1874.402337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.405956] Lustre: Skipped 3 previous similar messages [ 1875.749544] LustreError: 102669:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1875.752794] LustreError: 102669:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1875.755880] LustreError: 98704:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1875.792321] LustreError: 102676:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -17 [ 1875.795818] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1875.795818] [ 1876.090208] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1879.532225] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1880.457848] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 01:26:36 (1759814796) [ 1888.392289] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1896.825873] Lustre: 104561:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1907.623312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1909.245568] Lustre: 105694:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1971.682300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.683805] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.684493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.684501] Lustre: Skipped 1 previous similar message [ 1971.698200] Lustre: Skipped 3 previous similar messages [ 1972.886753] LustreError: 106153:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1972.891223] LustreError: 106153:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1972.930527] Lustre: server umount lustre-MDT0000 complete [ 1974.560704] LustreError: 99438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814891 with bad export cookie 14414773951690565645 [ 1974.562057] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.566113] LustreError: 99438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1974.757667] Lustre: server umount lustre-MDT0001 complete [ 1987.016803] Lustre: server umount lustre-OST0000 complete [ 1998.990408] Lustre: server umount lustre-OST0001 complete [ 2001.213042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2004.941508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2005.364581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2008.865478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2012.972949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2013.386608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2022.106996] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2026.885713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.891618] Lustre: lustre-MDT0000: reset Object Index mappings [ 2027.086131] LustreError: 109632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.117712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.805344] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2032.493194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2032.501963] Lustre: lustre-MDT0001: reset Object Index mappings [ 2032.609361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2032.611822] Lustre: Skipped 2 previous similar messages [ 2034.539528] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2035.777087] Lustre: 110748:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2038.469723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2041.243982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2042.855451] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2044.890689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2047.222725] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2047.640507] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2051.309449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2051.309537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2056.720904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2058.327447] Lustre: 112601:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2082.180502] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:29:58 (1759814998) [ 2112.994608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.997350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.999310] Lustre: Skipped 3 previous similar messages [ 2113.003016] Lustre: Skipped 2 previous similar messages [ 2117.390237] LustreError: 112950:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2117.392180] LustreError: 112950:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2117.432901] Lustre: server umount lustre-MDT0000 complete [ 2118.112219] LustreError: 112621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.119113] LustreError: 112621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2118.876520] LustreError: 109612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815035 with bad export cookie 14414773951690612685 [ 2118.881378] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.881883] LustreError: 109612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2123.232751] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.236349] Lustre: Skipped 1 previous similar message [ 2125.295567] Lustre: server umount lustre-MDT0001 complete [ 2141.151095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2141.197498] Lustre: server umount lustre-OST0000 complete [ 2157.023098] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2157.118396] Lustre: server umount lustre-OST0001 complete [ 2159.635590] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2163.808859] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2168.187277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2170.982849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2173.412292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2176.098501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2178.924420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2178.953735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.053917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2179.069618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2179.116485] Lustre: lustre-MDT0000: new disk, initializing [ 2179.149988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.152045] Lustre: Skipped 3 previous similar messages [ 2179.158675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2180.763579] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2184.185561] LustreError: 116304:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2184.187811] LustreError: 116304:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2184.274519] Lustre: server umount lustre-MDT0000 complete [ 2185.346467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.877908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.906421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.021148] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2188.028227] Lustre: 116825:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2188.033648] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2189.696393] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2191.678458] Lustre: server umount lustre-MDT0000 complete [ 2195.136824] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:31:51 (1759815111) [ 2195.838704] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2196.595559] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:31:52 (1759815112) [ 2203.365256] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2207.570111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.385987] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2212.636235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2212.665147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2212.701257] Lustre: 118841:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2212.726744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2212.728798] Lustre: Skipped 1 previous similar message [ 2212.767900] Lustre: lustre-MDT0001: new disk, initializing [ 2212.831799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2212.835410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2214.389630] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2216.864331] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2219.544360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2219.571639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2219.691977] Lustre: lustre-OST0000: new disk, initializing [ 2219.695062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2220.480231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2220.485235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2220.534597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2222.097903] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2227.200596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2227.231274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2229.282628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2229.288323] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2229.339486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2230.143522] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2235.525336] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2237.151288] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2393.794195] Lustre: 127609:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2393.799090] Lustre: 127609:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2393.802608] Lustre: 127609:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2393.805509] Lustre: 127609:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2393.809649] Lustre: 127609:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2393.814327] Lustre: 127609:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2498.598846] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:36:54 (1759815414) [ 2510.432552] LustreError: 127820:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2510.434724] LustreError: 127820:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2510.504781] Lustre: server umount lustre-MDT0000 complete [ 2510.818299] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.819258] LustreError: 122031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.822301] Lustre: Skipped 3 previous similar messages [ 2510.830413] LustreError: 122031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2511.871425] LustreError: 118833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815428 with bad export cookie 14414773951690668692 [ 2511.875438] LustreError: 118833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2511.876650] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.939625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2515.942757] Lustre: Skipped 4 previous similar messages [ 2518.147108] Lustre: server umount lustre-MDT0001 complete [ 2525.629262] Lustre: server umount lustre-OST0000 complete [ 2533.180800] Lustre: server umount lustre-OST0001 complete [ 2535.630570] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2539.339400] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2543.374536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2545.858115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2548.238853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2550.692893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2553.257204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2553.282111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.385175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2553.398828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2553.400873] Lustre: Skipped 1 previous similar message [ 2553.440290] Lustre: lustre-MDT0000: new disk, initializing [ 2553.441933] Lustre: Skipped 1 previous similar message [ 2553.470499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.473477] Lustre: Skipped 5 previous similar messages [ 2553.480941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2554.931148] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2559.277987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.304929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2559.337529] Lustre: 130607:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2559.461560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2560.835279] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2564.107982] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2565.474432] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2567.935556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.961990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2569.150706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2569.187035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2570.197996] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2573.485065] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2576.998843] Lustre: lustre-MDD0000: changelog on [ 2578.066307] Lustre: lustre-MDD0001: changelog on [ 2975.178871] Lustre: lustre-MDD0001: changelog off [ 2975.800683] Lustre: 130612:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2976.119497] Lustre: lustre-MDD0000: changelog off [ 2978.786397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2978.787216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.788085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2978.788091] Lustre: Skipped 2 previous similar messages [ 2978.790093] LustreError: Skipped 1 previous similar message [ 2978.796090] Lustre: Skipped 3 previous similar messages [ 2983.724138] LustreError: 139849:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2983.726292] LustreError: 139849:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2983.820465] Lustre: server umount lustre-OST0000 complete [ 2991.370959] Lustre: server umount lustre-MDT0000 complete [ 2992.713850] LustreError: 130597:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815909 with bad export cookie 14414773951691472957 [ 2992.717540] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2992.718349] LustreError: 130597:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2992.869928] Lustre: server umount lustre-MDT0001 complete [ 2997.043780] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2998.240807] Key type lgssc unregistered [ 2998.395480] LNet: 140783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2998.398355] LNetError: 140783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2998.407470] LNet: Removed LNI 192.168.201.121@tcp [ 2998.770176] Key type .llcrypt unregistered [ 2998.772058] Key type ._llcrypt unregistered [ 3005.815607] Key type ._llcrypt registered [ 3005.817019] Key type .llcrypt registered [ 3005.860818] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3012.817441] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3013.212249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3013.232398] alg: No test for adler32 (adler32-zlib) [ 3014.093493] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3014.189497] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3015.783178] Key type lgssc registered [ 3016.183829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3019.768881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3022.377906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.721392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3027.144831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.739668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.762488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3029.769184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.868469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3030.881944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3030.921535] Lustre: lustre-MDT0000: new disk, initializing [ 3030.948854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.959529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3032.458422] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3037.023378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3037.051333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3037.093679] Lustre: 143931:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3037.123313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3037.126110] Lustre: Skipped 1 previous similar message [ 3037.164503] Lustre: lustre-MDT0001: new disk, initializing [ 3037.203587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3037.216465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3037.220281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3038.655675] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3041.983627] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3043.314486] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3045.850808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3045.876193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3045.983986] Lustre: lustre-OST0000: new disk, initializing [ 3045.986517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3046.017952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.651163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3046.655283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3046.707063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3048.160341] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3051.555456] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3053.924161] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_module obdecho/obdecho [ 3054.602986] LustreError: 146560:0:(echo_client.c:669:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3054.605841] LustreError: 146560:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 3055.267942] LustreError: 146656:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 3062.241076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.242377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.246054] Lustre: Skipped 2 previous similar messages [ 3062.249733] Lustre: Skipped 2 previous similar messages [ 3065.021318] LustreError: 146995:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3065.053214] Lustre: server umount lustre-MDT0000 complete [ 3066.330692] LustreError: 143923:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815983 with bad export cookie 14611442450014123462 [ 3066.333892] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3066.333982] LustreError: 143923:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3066.413836] LustreError: 147197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3066.416980] LustreError: 147197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3066.503265] Lustre: server umount lustre-MDT0001 complete [ 3078.130485] LustreError: 147398:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3078.133622] LustreError: 147398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3078.205256] Lustre: server umount lustre-OST0000 complete [ 3085.503431] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3089.192730] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3092.628595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3094.962030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3097.124114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.330787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.679914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.703745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.784597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3101.796573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3101.834616] Lustre: lustre-MDT0000: new disk, initializing [ 3101.858635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.864587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3103.208365] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3107.587420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3107.612527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3107.647298] Lustre: 150338:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3107.671100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3107.673174] Lustre: Skipped 1 previous similar message [ 3107.710315] Lustre: lustre-MDT0001: new disk, initializing [ 3107.745028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3107.757551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3107.760627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3109.147605] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3112.403419] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3113.742873] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3116.141905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3116.166913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3116.265768] Lustre: lustre-OST0000: new disk, initializing [ 3116.267663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3116.295843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.357848] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3119.289351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3119.293170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3119.327937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3121.693417] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3123.215066] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3123.302513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3124.776721] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3124.860927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3129.826146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.826192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3129.827168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3129.831455] Lustre: Skipped 1 previous similar message [ 3129.837361] Lustre: Skipped 1 previous similar message [ 3131.627086] LustreError: 153276:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3131.629492] LustreError: 153276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3131.690362] Lustre: server umount lustre-OST0000 complete [ 3133.408052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3133.410862] LustreError: Skipped 1 previous similar message [ 3133.412139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.417338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.135336] LustreError: 153478:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3139.138512] LustreError: 153478:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3139.177280] Lustre: server umount lustre-MDT0000 complete [ 3140.465301] LustreError: 150329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816057 with bad export cookie 14611442450014124407 [ 3140.468103] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.468882] LustreError: 150329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3140.613720] Lustre: server umount lustre-MDT0001 complete [ 3141.835796] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:47:37 (1759816057) [ 3142.467191] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3143.154649] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:47:39 (1759816059) [ 3145.902304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.054594] LustreError: 154348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.060745] LustreError: 154348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3146.078353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.425878] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3150.378680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3151.889510] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3153.805498] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3155.169119] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3157.660276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3159.760626] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3161.777846] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3176.418790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.420181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3176.424498] Lustre: Skipped 3 previous similar messages [ 3176.426370] Lustre: Skipped 3 previous similar messages [ 3176.430508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3180.352365] LustreError: 156612:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3180.354583] LustreError: 156612:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3180.388171] Lustre: server umount lustre-MDT0000 complete [ 3181.537116] LustreError: 154366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.542752] LustreError: 154366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3181.715263] LustreError: 154327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816098 with bad export cookie 14611442450014125275 [ 3181.718815] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.719056] LustreError: 154327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3182.360528] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.21@tcp (stopping) [ 3182.363735] Lustre: Skipped 2 previous similar messages [ 3186.656301] LustreError: 156419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.656607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.662275] LustreError: 156419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3191.776611] LustreError: 155633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.777195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3191.782515] LustreError: 155633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3191.785608] Lustre: Skipped 2 previous similar messages [ 3195.871188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3195.988675] Lustre: server umount lustre-MDT0001 complete [ 3198.267890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3202.681512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3202.758283] LustreError: 157874:0:(llog.c:1602:llog_backup()) MGC192.168.201.121@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3202.761189] Lustre: 157874:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.121@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3216.351862] LustreError: 157882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.357289] LustreError: 157882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3226.592102] LustreError: 157881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.597480] LustreError: 157881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3242.766987] LustreError: 157901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.771961] LustreError: 157901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 3263.967385] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3263.969649] Lustre: 157874:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3277.279942] LustreError: 157901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.285527] LustreError: 157901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3290.079323] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3290.081401] Lustre: 154339:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3304.487627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3304.491521] Lustre: Skipped 2 previous similar messages [ 3305.869689] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3307.490817] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3308.818580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.559288] LustreError: 154339:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3327.970560] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcac6457e5927d63b [ 3327.973428] Lustre: MGC192.168.201.121@tcp: Connection restored to 0@lo (at 0@lo) [ 3328.041652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 3329.441440] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3331.487194] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3332.153839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3333.096685] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3333.566602] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3339.301356] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 5 sec [ 3346.301633] LustreError: 159363:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3346.303621] LustreError: 159363:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3346.335576] Lustre: server umount lustre-MDT0000 complete [ 3347.650325] LustreError: 154328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816264 with bad export cookie 14611442450014131771 [ 3347.654217] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3347.655705] LustreError: 154328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3347.802532] Lustre: server umount lustre-MDT0001 complete [ 3368.633310] Lustre: server umount lustre-OST0000 complete [ 3376.265824] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3380.257192] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3383.797477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3386.376884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3388.674691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.060222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.590183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.617736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.712305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3393.729132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3393.774451] Lustre: lustre-MDT0000: new disk, initializing [ 3393.805257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3393.808497] Lustre: Skipped 1 previous similar message [ 3393.816260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3395.279971] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3399.669323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3399.694705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.730471] Lustre: 162708:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3399.835358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3401.308852] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3404.642902] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3406.086788] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3408.488060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3408.514919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3410.803963] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3413.914344] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3415.187212] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3418.099333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3418.128833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3418.335709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3419.754385] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3419.859359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3423.202490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.202820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.204662] LustreError: Skipped 1 previous similar message [ 3423.211683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.215710] Lustre: Skipped 2 previous similar messages [ 3426.670141] LustreError: 165667:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3426.673057] LustreError: 165667:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3426.742268] Lustre: server umount lustre-OST0000 complete [ 3434.195061] Lustre: server umount lustre-MDT0000 complete [ 3435.493308] LustreError: 162701:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816352 with bad export cookie 14611442450014148564 [ 3435.496150] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.498166] LustreError: 162701:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3435.633862] Lustre: server umount lustre-MDT0001 complete [ 3438.785077] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:52:34 (1759816354) [ 3441.066175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.263396] LustreError: 166740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.272052] LustreError: 166740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 3442.633906] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3445.544290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3447.102740] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3448.937165] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3450.204522] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3452.546446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3454.622142] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3456.501066] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3471.227181] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:53:07 (1759816387) [ 3472.353798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.354265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3472.359832] Lustre: Skipped 3 previous similar messages [ 3472.362984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3472.366140] Lustre: Skipped 5 previous similar messages [ 3478.436188] Lustre: server umount lustre-OST0000 complete [ 3494.367125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3494.410414] LustreError: 169400:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3494.412550] LustreError: 169400:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3494.440310] Lustre: server umount lustre-MDT0000 complete [ 3495.812422] LustreError: 166720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816412 with bad export cookie 14611442450014149432 [ 3495.814943] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.817190] LustreError: 166720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3495.959103] Lustre: server umount lustre-MDT0001 complete [ 3500.166170] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3501.424300] Key type lgssc unregistered [ 3501.567448] LNet: 170135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3501.571439] LNetError: 170135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3501.583380] LNet: Removed LNI 192.168.201.121@tcp [ 3501.971845] Key type .llcrypt unregistered [ 3501.973052] Key type ._llcrypt unregistered [ 3510.399664] Key type ._llcrypt registered [ 3510.401265] Key type .llcrypt registered [ 3510.444686] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3510.777457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3510.784506] alg: No test for adler32 (adler32-zlib) [ 3511.648996] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3511.738713] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3513.319111] Key type lgssc registered [ 3513.696132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3516.529551] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3516.535901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3531.999447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3537.119193] LustreError: 171350:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 3542.239373] LustreError: 171350:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3542.243067] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3542.244752] LustreError: MGC192.168.201.121@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3542.247616] LustreError: 171350:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3542.249980] LustreError: 171350:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 3542.252201] LustreError: 171350:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3542.254235] LustreError: 171350:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3542.302449] Lustre: server umount lustre-OST0000 complete [ 3542.303710] LustreError: 171350:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3545.472751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.641083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.092350] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3551.259149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3551.388690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3552.820808] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3554.890388] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3556.304671] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3558.853551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3558.937244] LustreError: 173441:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3558.943317] Lustre: 173441:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.121@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3559.001533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.245288] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3563.224901] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3567.072484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3567.075362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.079729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3567.587205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.588458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3567.591699] Lustre: Skipped 1 previous similar message [ 3567.594946] Lustre: Skipped 1 previous similar message [ 3572.704781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3572.707862] Lustre: Skipped 2 previous similar messages [ 3577.826910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.829174] Lustre: Skipped 2 previous similar messages [ 3578.335167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3578.376693] LustreError: 174088:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3578.378547] LustreError: 174088:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3578.406570] Lustre: server umount lustre-MDT0000 complete [ 3579.733364] LustreError: 171831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816496 with bad export cookie 5916971185251484976 [ 3579.736745] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3579.745082] LustreError: 171831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3579.807752] LustreError: 174295:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3579.810856] LustreError: 174295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3579.898773] Lustre: server umount lustre-MDT0001 complete [ 3591.537509] LustreError: 174496:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3591.539441] LustreError: 174496:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3591.611128] Lustre: server umount lustre-OST0000 complete [ 3594.993462] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3598.876666] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3602.569316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.934451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3607.280916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.747812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3612.282214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3612.308316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.395950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3612.409476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3612.447636] Lustre: lustre-MDT0000: new disk, initializing [ 3612.472884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.480721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3613.987337] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3618.494921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3618.520039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3618.560058] Lustre: 176831:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3618.586272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3618.588766] Lustre: Skipped 1 previous similar message [ 3618.626552] Lustre: lustre-MDT0001: new disk, initializing [ 3618.664520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3618.677354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3618.682232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3620.120254] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3623.471345] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3624.853043] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3627.372052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.397432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3627.504755] Lustre: lustre-OST0000: new disk, initializing [ 3627.507776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3627.548779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.590511] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3632.823848] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3634.168543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3634.172447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3634.211545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3634.308135] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3634.390702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3635.851861] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3635.932775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3639.265039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3639.265362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.268534] LustreError: Skipped 1 previous similar message [ 3639.275459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3639.278561] Lustre: Skipped 1 previous similar message [ 3642.738105] LustreError: 179768:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3642.741059] LustreError: 179768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3642.808452] Lustre: server umount lustre-OST0000 complete [ 3644.384631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3644.384969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.394358] Lustre: Skipped 2 previous similar messages [ 3649.504308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.506698] Lustre: Skipped 3 previous similar messages [ 3650.330219] Lustre: server umount lustre-MDT0000 complete [ 3651.698308] LustreError: 176823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816568 with bad export cookie 5916971185251485662 [ 3651.702632] LustreError: 176823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.702968] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.766384] LustreError: 180172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3651.769551] LustreError: 180172:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3651.857493] Lustre: server umount lustre-MDT0001 complete [ 3654.993726] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:56:10 (1759816570) [ 3661.324746] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3662.469442] Key type lgssc unregistered [ 3662.605320] LNet: 181353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3662.609336] LNetError: 181353:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3662.616350] LNet: Removed LNI 192.168.201.121@tcp [ 3662.948971] Key type .llcrypt unregistered [ 3662.949936] Key type ._llcrypt unregistered [ 3670.570837] Key type ._llcrypt registered [ 3670.572395] Key type .llcrypt registered [ 3670.614663] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3670.990329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3671.039179] alg: No test for adler32 (adler32-zlib) [ 3671.919256] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3672.009673] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3673.599183] Key type lgssc registered [ 3673.996536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.838053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.846036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.013622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.467380] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3682.583625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3682.719445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3684.170212] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3686.022734] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3687.326332] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3689.968864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3690.046624] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3690.048096] LustreError: 184167:0:(llog.c:1642:llog_backup()) MGC192.168.201.121@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3690.051800] Lustre: 184167:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.121@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3690.094324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.082257] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3693.924704] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3698.143893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3698.146864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.151923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.658342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.658815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.664118] Lustre: Skipped 1 previous similar message [ 3700.798090] LustreError: 184820:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3700.825679] Lustre: server umount lustre-MDT0000 complete [ 3702.090137] LustreError: 182508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816618 with bad export cookie 5968052466138748998 [ 3702.093181] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3702.094297] LustreError: 182508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3702.154980] LustreError: 185023:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3702.157763] LustreError: 185023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3702.232725] Lustre: server umount lustre-MDT0001 complete [ 3713.777464] LustreError: 185224:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3713.779523] LustreError: 185224:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3713.842880] Lustre: server umount lustre-OST0000 complete [ 3716.988601] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3720.657612] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3724.434092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3726.879425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3729.095735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3731.344377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3733.743079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3733.773255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.851204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3733.866134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3733.905116] Lustre: lustre-MDT0000: new disk, initializing [ 3733.930045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.936814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3735.345361] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3739.736555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3739.757451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3739.794153] Lustre: 187556:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3739.816698] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3739.818200] Lustre: Skipped 1 previous similar message [ 3739.819749] LustreError: 188321:0:(llog.c:1642:llog_backup()) MGC192.168.201.121@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3739.823655] LustreError: 188321:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3739.825880] Lustre: 188321:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.121@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3739.833950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3739.835858] Lustre: Skipped 1 previous similar message [ 3739.874322] Lustre: lustre-MDT0001: new disk, initializing [ 3739.907833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3739.921357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3739.925355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3741.294760] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3744.486677] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3745.784187] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3748.201941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3748.225522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3748.323945] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3748.325110] Lustre: Skipped 1 previous similar message [ 3748.326353] LustreError: 189276:0:(llog.c:1642:llog_backup()) MGC192.168.201.121@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3748.329045] LustreError: 189276:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3748.330727] Lustre: 189276:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.121@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3748.333315] Lustre: 189276:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3748.341349] Lustre: lustre-OST0000: new disk, initializing [ 3748.343257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3748.370440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.300793] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3752.952324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3752.956234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3752.989173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3753.578127] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3755.107828] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3755.206957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3756.659768] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3756.747512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3758.048622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3758.049747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.051206] LustreError: Skipped 1 previous similar message [ 3758.056086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.058934] Lustre: Skipped 2 previous similar messages [ 3759.469125] LustreError: 190495:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3759.471620] LustreError: 190495:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3759.534112] Lustre: server umount lustre-OST0000 complete [ 3763.168787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.169637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.175670] Lustre: Skipped 2 previous similar messages [ 3766.908872] LustreError: 190695:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3766.911626] LustreError: 190695:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3766.940691] Lustre: server umount lustre-MDT0000 complete [ 3768.210197] LustreError: 187547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816684 with bad export cookie 5968052466138749698 [ 3768.213371] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3768.215187] LustreError: 187547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3768.371951] Lustre: server umount lustre-MDT0001 complete [ 3771.326424] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:58:07 (1759816687) [ 3776.875584] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3780.328493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.490235] LustreError: 192264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.495938] LustreError: 192264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3780.508327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.897457] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3784.795773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3786.237400] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3787.238204] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3789.510436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3791.538226] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3794.501906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3794.528170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3794.584303] Lustre: lustre-OST0001: new disk, initializing [ 3794.586181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3796.616939] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3800.066918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3800.070474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3800.106378] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3802.065647] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3808.826575] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3814.486969] Lustre: DEBUG MARKER: ADD OST3 [ 3816.197413] loop: module loaded [ 3816.216095] loop0: detected capacity change from 0 to 204800000 [ 3816.337018] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3816.340509] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3816.344631] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3816.396382] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3818.451087] loop0: detected capacity change from 0 to 204800000 [ 3820.342026] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3820.369322] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3820.441558] Lustre: lustre-OST0003: new disk, initializing [ 3820.444173] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3820.495088] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3820.497516] Lustre: Skipped 3 previous similar messages [ 3821.348371] Lustre: DEBUG MARKER: STOP OST3 [ 3822.064878] Lustre: Failing over lustre-OST0003 [ 3822.093490] LustreError: 196606:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3822.095606] LustreError: 196606:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3822.116700] Lustre: server umount lustre-OST0003 complete [ 3822.239493] LustreError: 193716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.244697] LustreError: 193716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3822.731904] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3825.295420] LustreError: 193717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.301750] LustreError: 193717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3826.729975] Lustre: DEBUG MARKER: START OST3 again [ 3827.343467] LustreError: 193717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3829.135736] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3830.453669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3830.458526] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3830.472095] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3831.202851] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3844.745611] Lustre: Failing over lustre-OST0003 [ 3844.767068] LustreError: 198558:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 3844.769024] LustreError: 198558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3844.783862] Lustre: server umount lustre-OST0003 complete [ 3846.112916] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.113684] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.115734] LustreError: Skipped 1 previous similar message [ 3846.120267] LustreError: 193716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.126565] LustreError: 193716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3854.820482] LustreError: 193718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3854.827442] LustreError: 193718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3859.938972] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.940257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.945607] Lustre: Skipped 4 previous similar messages [ 3859.948733] Lustre: Skipped 4 previous similar messages [ 3863.391429] Lustre: server umount lustre-MDT0000 complete [ 3864.619169] LustreError: 192243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816781 with bad export cookie 5968052466138750566 [ 3864.623146] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3864.623212] LustreError: 192243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3869.088257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3869.090595] Lustre: Skipped 3 previous similar messages [ 3870.863266] Lustre: server umount lustre-MDT0001 complete [ 3872.231282] Lustre: server umount lustre-OST0000 complete [ 3873.541036] Lustre: server umount lustre-OST0001 complete [ 3875.660377] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3878.563376] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3882.190108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3884.666717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3886.845763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3889.100738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3891.493149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3891.524153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.611166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3891.624256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3891.663856] Lustre: lustre-MDT0000: new disk, initializing [ 3891.691785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.693987] Lustre: Skipped 1 previous similar message [ 3891.699973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3893.145660] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3897.542773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3897.566775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3897.603951] Lustre: 201613:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3897.717895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3899.028242] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3902.278583] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3903.533828] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3905.887403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3905.912808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3907.393350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3907.441853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3908.072337] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3911.131826] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3912.556572] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3912.647923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3914.085284] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3914.177561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3917.794331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.796101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3917.797825] Lustre: Skipped 2 previous similar messages [ 3917.800435] LustreError: Skipped 1 previous similar message [ 3917.801102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.805007] Lustre: Skipped 1 previous similar message [ 3929.055203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3929.072773] LustreError: 204550:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3929.075041] LustreError: 204550:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3929.139470] Lustre: server umount lustre-OST0000 complete [ 3936.533713] Lustre: server umount lustre-MDT0000 complete [ 3937.835440] LustreError: 202391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816854 with bad export cookie 5968052466138752827 [ 3937.839936] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.002847] Lustre: server umount lustre-MDT0001 complete [ 3939.321249] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:00:55 (1759816855) [ 3944.732599] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3948.211414] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3951.776079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3954.087800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3956.306609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3958.509252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.802520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.827241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.909521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3960.920944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3960.922668] Lustre: Skipped 3 previous similar messages [ 3960.957515] Lustre: lustre-MDT0000: new disk, initializing [ 3960.958786] Lustre: Skipped 2 previous similar messages [ 3960.988445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3960.991696] Lustre: Skipped 2 previous similar messages [ 3962.264777] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3966.355692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.378224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3966.411297] Lustre: 207806:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3966.523770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3967.791281] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3970.941248] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3972.139277] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3974.381777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3974.401819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3975.796931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3976.508331] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3979.616959] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3980.975439] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3981.048314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3982.329451] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3982.414229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3985.130957] Lustre: server umount lustre-OST0000 complete [ 3986.400211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.400888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.405773] Lustre: Skipped 3 previous similar messages [ 3986.407444] Lustre: Skipped 6 previous similar messages [ 3996.644822] LustreError: 210945:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3996.646865] LustreError: 210945:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3996.673314] Lustre: server umount lustre-MDT0000 complete [ 3997.905310] LustreError: 208587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816914 with bad export cookie 5968052466138753709 [ 3997.908995] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3997.909028] LustreError: 208587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4007.471536] Lustre: server umount lustre-MDT0001 complete [ 4010.287950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4010.433179] LustreError: 211625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4010.440289] LustreError: 211625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4011.731644] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4014.555733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4015.995162] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4017.778943] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4018.986640] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4021.297626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4021.414385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.416469] Lustre: Skipped 7 previous similar messages [ 4023.393378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4025.243247] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4045.280439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4048.166884] Lustre: server umount lustre-OST0000 complete [ 4050.400234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.406804] Lustre: Skipped 2 previous similar messages [ 4050.409568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.411555] Lustre: Skipped 8 previous similar messages [ 4055.518846] Lustre: server umount lustre-MDT0000 complete [ 4056.544559] LustreError: 212947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.550061] LustreError: 212947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4056.823369] LustreError: 211605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816973 with bad export cookie 5968052466138754591 [ 4056.826875] LustreError: 211605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4056.827936] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4056.969200] Lustre: server umount lustre-MDT0001 complete [ 4158.425523] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4159.591342] Key type lgssc unregistered [ 4159.737528] LNet: 214961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4159.741043] LNetError: 214961:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4159.748439] LNet: Removed LNI 192.168.201.121@tcp [ 4160.079129] Key type .llcrypt unregistered [ 4160.080776] Key type ._llcrypt unregistered [ 4168.443602] Key type ._llcrypt registered [ 4168.444720] Key type .llcrypt registered [ 4168.482732] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4174.823492] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4175.147720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4175.206092] alg: No test for adler32 (adler32-zlib) [ 4176.062362] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4176.143142] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4177.719165] Key type lgssc registered [ 4178.072674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.950582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.236907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4185.225222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4187.367816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4189.616739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4189.634824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4189.639588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.720553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4190.733283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4190.771994] Lustre: lustre-MDT0000: new disk, initializing [ 4190.796699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4190.803886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4192.116165] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4196.256414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4196.280037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4196.312314] Lustre: 218260:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4196.336966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4196.339528] Lustre: Skipped 1 previous similar message [ 4196.381106] Lustre: lustre-MDT0001: new disk, initializing [ 4196.414497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.426038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4196.430990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4197.688754] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4200.747776] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4201.943275] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4204.262247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4204.286363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4204.389859] Lustre: lustre-OST0000: new disk, initializing [ 4204.391701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4204.419470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.359769] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4206.840519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4206.843801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4206.879660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4209.447486] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4210.784544] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4210.858818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4212.146459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4212.227013] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4214.893631] LustreError: 221202:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4214.967921] Lustre: server umount lustre-OST0000 complete [ 4216.800127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4216.803370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.809322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.312541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.312937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.432576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.435736] Lustre: Skipped 2 previous similar messages [ 4225.507418] LustreError: 221403:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4225.509697] LustreError: 221403:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4225.535642] Lustre: server umount lustre-MDT0000 complete [ 4226.804323] LustreError: 220008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817143 with bad export cookie 14702426435087372704 [ 4226.807140] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.807919] LustreError: 220008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4237.283810] LustreError: 221604:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4237.286018] LustreError: 221604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4237.367982] Lustre: server umount lustre-MDT0001 complete [ 4243.514123] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4246.906355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.065478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.558504] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4251.716834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4251.842164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4253.152891] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4254.132915] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4256.403537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4258.465947] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4261.286726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4261.310967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4261.366833] Lustre: lustre-OST0001: new disk, initializing [ 4261.368847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4261.405005] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4261.407418] Lustre: Skipped 1 previous similar message [ 4262.525243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4262.528478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4262.560360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4263.277721] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4268.767687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4273.451342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4282.463186] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:06:38 (1759817198) [ 4283.363277] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.366364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.369439] Lustre: Skipped 3 previous similar messages [ 4283.371559] Lustre: Skipped 2 previous similar messages [ 4288.482586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4288.485381] Lustre: Skipped 3 previous similar messages [ 4289.275128] LustreError: 226171:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4289.277236] LustreError: 226171:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4289.303927] Lustre: server umount lustre-MDT0000 complete [ 4290.564167] LustreError: 225809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817207 with bad export cookie 14702426435087373572 [ 4290.567811] LustreError: 225809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4290.568416] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4290.717666] Lustre: server umount lustre-MDT0001 complete [ 4301.812127] LustreError: 226573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4301.814718] LustreError: 226573:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4301.830835] Lustre: server umount lustre-OST0000 complete [ 4312.304915] LustreError: 226775:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4312.307107] LustreError: 226775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4312.372192] Lustre: server umount lustre-OST0001 complete [ 4314.547561] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4317.840110] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4321.195817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4323.400113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4325.518703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4327.748685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4330.057970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4330.088819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.173082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4330.184099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4330.220631] Lustre: lustre-MDT0000: new disk, initializing [ 4330.243562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.250611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4331.541716] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4335.681372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4335.704584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4335.737672] Lustre: 228957:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4335.837854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4337.164079] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4340.376053] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4341.716978] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4343.998763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4344.022075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4344.119427] Lustre: lustre-OST0000: new disk, initializing [ 4344.120750] Lustre: Skipped 1 previous similar message [ 4344.122521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4344.124889] Lustre: Skipped 2 previous similar messages [ 4345.594315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4345.598308] Lustre: Skipped 1 previous similar message [ 4345.600746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4345.636632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4346.025862] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4349.168577] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4350.535025] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4350.612261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4351.967904] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4352.061869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4356.066620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4356.068375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.070494] LustreError: Skipped 1 previous similar message [ 4356.074647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4356.078217] Lustre: Skipped 1 previous similar message [ 4358.830994] LustreError: 231895:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4358.833382] LustreError: 231895:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4358.912649] Lustre: server umount lustre-OST0000 complete [ 4361.185458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.191880] Lustre: Skipped 2 previous similar messages [ 4366.362952] Lustre: server umount lustre-MDT0000 complete [ 4367.628793] LustreError: 229736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817284 with bad export cookie 14702426435087375672 [ 4367.631966] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.635568] LustreError: 229736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4367.779042] Lustre: server umount lustre-MDT0001 complete [ 4370.634853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.806850] LustreError: 232772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.814587] LustreError: 232772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4370.835267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.838627] Lustre: Skipped 2 previous similar messages [ 4372.186128] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4375.075988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4376.513845] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4378.315392] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4379.550230] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4381.823543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4383.790534] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4385.598984] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4528.607876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4528.610691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.615348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4528.617381] Lustre: Skipped 4 previous similar messages [ 4534.316259] LustreError: 235057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4534.319525] LustreError: 235057:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4534.382425] Lustre: server umount lustre-OST0000 complete [ 4539.360484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.363372] LustreError: Skipped 1 previous similar message [ 4550.111148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4550.169898] Lustre: server umount lustre-MDT0000 complete [ 4551.435582] LustreError: 233493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817468 with bad export cookie 14702426435087376554 [ 4551.439185] LustreError: 233493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4551.439333] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.607615] Lustre: server umount lustre-MDT0001 complete [ 4649.923392] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4650.953408] Key type lgssc unregistered [ 4651.078384] LNet: 236030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4651.081097] LNetError: 236030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4651.091276] LNet: Removed LNI 192.168.201.121@tcp [ 4651.377102] Key type .llcrypt unregistered [ 4651.378233] Key type ._llcrypt unregistered [ 4659.229737] Key type ._llcrypt registered [ 4659.230813] Key type .llcrypt registered [ 4659.266842] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4665.133833] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4665.524686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4665.541427] alg: No test for adler32 (adler32-zlib) [ 4666.380276] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4666.453938] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4668.039096] Key type lgssc registered [ 4668.352636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4671.057021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4673.117819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4675.066116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4677.128690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4682.552635] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4685.820032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.839723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4685.845814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.929039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4686.941318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4686.981459] Lustre: lustre-MDT0000: new disk, initializing [ 4687.004282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4687.010201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4688.228697] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4692.187702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4692.209803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4692.238460] Lustre: 240026:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4692.259198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4692.260949] Lustre: Skipped 1 previous similar message [ 4692.296840] Lustre: lustre-MDT0001: new disk, initializing [ 4692.326809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4692.337266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4692.340310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4693.538665] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4695.760364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4697.959772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4697.988912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4698.094373] Lustre: lustre-OST0000: new disk, initializing [ 4698.096246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4698.133557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.399831] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4703.736722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4703.739526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4703.773794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4705.046870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4705.080428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4705.144374] Lustre: lustre-OST0001: new disk, initializing [ 4705.147071] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4705.178659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4707.077436] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4709.181114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4709.184222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4709.211667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4712.510225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4717.241265] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4728.287963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4728.290643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.294554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.825449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.827649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.835749] Lustre: Skipped 1 previous similar message [ 4731.590897] LustreError: 243553:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4731.625708] Lustre: server umount lustre-MDT0000 complete [ 4732.876768] LustreError: 240017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817649 with bad export cookie 16935847151139384397 [ 4732.881411] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4732.945678] LustreError: 243754:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4732.948114] LustreError: 243754:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4733.034997] Lustre: server umount lustre-MDT0001 complete [ 4745.143756] LustreError: 243955:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4745.146086] LustreError: 243955:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4745.168227] Lustre: server umount lustre-OST0000 complete [ 4756.594473] LustreError: 244157:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4756.596399] LustreError: 244157:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4756.665236] Lustre: server umount lustre-OST0001 complete [ 4761.097914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4764.053362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.399530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4767.161825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4770.018481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4770.360906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4776.357777] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4779.809801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.815276] Lustre: lustre-MDT0000: reset Object Index mappings [ 4779.979133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.294094] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4784.057615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4784.062477] Lustre: lustre-MDT0001: reset Object Index mappings [ 4785.418987] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4786.364626] Lustre: 248575:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4788.454087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4788.586100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.587876] Lustre: Skipped 1 previous similar message [ 4789.602208] LustreError: 248932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.605745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4789.607033] LustreError: 248932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4790.518289] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4793.563559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4794.669295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4795.680827] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4798.945097] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4800.205151] Lustre: 250421:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4809.695803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4809.698032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.701413] Lustre: Skipped 1 previous similar message [ 4809.702941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4809.705596] Lustre: Skipped 3 previous similar messages [ 4812.541043] LustreError: 250525:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4812.544402] LustreError: 250525:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4812.576462] Lustre: server umount lustre-MDT0000 complete [ 4813.755188] LustreError: 247443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817730 with bad export cookie 16935847151139391173 [ 4813.758402] LustreError: 247443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4813.759144] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4813.891811] Lustre: server umount lustre-MDT0001 complete [ 4824.498085] LustreError: 250926:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4824.500036] LustreError: 250926:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4824.513757] Lustre: server umount lustre-OST0000 complete [ 4835.956324] Lustre: server umount lustre-OST0001 complete [ 4842.124977] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:15:58 (1759817758) [ 4847.357310] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4850.692927] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4854.183092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.419599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4858.563886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.929534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.189922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.212831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.296320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4863.306543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4863.340536] Lustre: lustre-MDT0000: new disk, initializing [ 4863.366420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.368150] Lustre: Skipped 1 previous similar message [ 4863.373817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4864.624056] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4868.600787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4868.624229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4868.656101] Lustre: 254643:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4868.754455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4870.020150] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4873.100208] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4874.320390] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4876.594397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.624062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.739161] Lustre: lustre-OST0000: new disk, initializing [ 4876.740444] Lustre: Skipped 1 previous similar message [ 4876.742065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4876.743510] Lustre: Skipped 2 previous similar messages [ 4878.079356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4878.081930] Lustre: Skipped 1 previous similar message [ 4878.084591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4878.119993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4878.683396] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4881.703161] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4883.113735] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4883.196840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4884.535322] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4884.629875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4887.342133] LustreError: 257586:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4887.344181] LustreError: 257586:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4887.424767] Lustre: server umount lustre-OST0000 complete [ 4889.568278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4889.571345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.576226] Lustre: Skipped 3 previous similar messages [ 4889.579151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4889.581161] Lustre: Skipped 4 previous similar messages [ 4893.664625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.664861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.811518] Lustre: server umount lustre-MDT0000 complete [ 4900.059601] LustreError: 254636:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817816 with bad export cookie 16935847151139393266 [ 4900.064942] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.614867] Lustre: server umount lustre-MDT0001 complete [ 4915.859169] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4919.579063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.721908] LustreError: 259163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.736749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.738915] Lustre: Skipped 2 previous similar messages [ 4921.064674] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4924.010616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4925.535519] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4926.563246] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4928.861159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4931.044979] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4933.995786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4934.023634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4934.088650] Lustre: lustre-OST0001: new disk, initializing [ 4934.090855] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4935.182796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4935.187113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4935.234248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4936.220869] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4940.758856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4941.996439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4948.824578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4948.860741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4955.103947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4955.106916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.111236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4955.112956] Lustre: Skipped 3 previous similar messages [ 4956.736752] LustreError: 262665:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4956.738632] LustreError: 262665:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4956.770839] Lustre: server umount lustre-MDT0000 complete [ 4958.074501] LustreError: 259144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817874 with bad export cookie 16935847151139394148 [ 4958.077941] LustreError: 259144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4958.078797] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4958.247255] Lustre: server umount lustre-MDT0001 complete [ 4969.988679] Lustre: server umount lustre-OST0000 complete [ 4981.641668] Lustre: server umount lustre-OST0001 complete [ 4987.307505] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4991.087632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.227113] LustreError: 264443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.246544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.248914] Lustre: Skipped 3 previous similar messages [ 4992.635973] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4995.570842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4997.123459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4998.115349] Lustre: 265552:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5000.388336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5001.570207] LustreError: 265907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.577534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5002.553487] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5005.531403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5006.646137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5007.670676] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5010.881387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5012.172734] Lustre: 267405:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5019.124199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5019.142535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5022.482682] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:18:58 (1759817938) [ 5036.512277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5036.515764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.519628] Lustre: Skipped 3 previous similar messages [ 5036.522692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.524379] Lustre: Skipped 3 previous similar messages [ 5040.196831] LustreError: 268113:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5040.198811] LustreError: 268113:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5040.227480] Lustre: server umount lustre-MDT0000 complete [ 5041.449838] LustreError: 264424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817958 with bad export cookie 16935847151139400427 [ 5041.453626] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.455972] LustreError: 264424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5042.656321] LustreError: 264439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.664878] LustreError: 264439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5047.659268] Lustre: server umount lustre-MDT0001 complete [ 5055.101560] Lustre: server umount lustre-OST0000 complete [ 5062.450617] Lustre: server umount lustre-OST0001 complete [ 5064.591504] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5068.139431] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5072.063237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5074.452933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5076.565395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5078.785283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5084.373075] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5087.902349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5087.928480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.008257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5088.022398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5088.057273] Lustre: lustre-MDT0000: new disk, initializing [ 5088.086773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5089.413272] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5093.637804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5093.661750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5093.697278] Lustre: 271594:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5093.811430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5095.218515] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5099.765096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5099.786800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5101.228423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5101.807642] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5106.066077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5106.100540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5107.949636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5108.293289] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5112.999173] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5123.612199] Lustre: DEBUG MARKER: == conf-sanity test 157: clear lu cache via procfs ======= 02:20:39 (1759818039) [ 5128.927147] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5134.768240] Lustre: 276544:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5142.976052] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5155.545689] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 02:21:11 (1759818071) [ 5159.392686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.393239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.397380] Lustre: Skipped 7 previous similar messages [ 5159.400285] Lustre: Skipped 9 previous similar messages [ 5162.642933] Lustre: server umount lustre-MDT0000 complete [ 5163.887759] LustreError: 272376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818080 with bad export cookie 16935847151139408295 [ 5163.889583] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.890806] LustreError: 272376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5164.027460] Lustre: server umount lustre-MDT0001 complete [ 5175.539250] LustreError: 278857:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5175.542225] LustreError: 278857:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 5175.566785] Lustre: server umount lustre-OST0000 complete [ 5187.252215] Lustre: server umount lustre-OST0001 complete [ 5193.123296] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5196.512734] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5199.869330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.151585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5204.234913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5206.393911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5208.963154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5208.988191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.066640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5209.069634] Lustre: Skipped 2 previous similar messages [ 5209.081570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5209.083319] Lustre: Skipped 4 previous similar messages [ 5209.122278] Lustre: lustre-MDT0000: new disk, initializing [ 5209.124124] Lustre: Skipped 3 previous similar messages [ 5209.150930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.152877] Lustre: Skipped 7 previous similar messages [ 5209.159922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5209.162735] Lustre: Skipped 3 previous similar messages [ 5210.462541] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5214.635704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5214.659695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5214.689958] Lustre: 281898:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5214.693655] Lustre: 281898:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 5214.799733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.804283] Lustre: Skipped 2 previous similar messages [ 5216.144227] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5219.364094] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5220.643545] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5222.939269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5222.964639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5224.431883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5225.067521] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5228.134259] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5228.853577] Lustre: Mounted lustre-client [ 5229.174539] LustreError: 284404:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9c78468bb000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5229.196116] Lustre: Unmounted lustre-client [ 5234.146210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.147438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.149870] Lustre: Skipped 2 previous similar messages [ 5234.154020] Lustre: Skipped 3 previous similar messages [ 5238.207411] Lustre: server umount lustre-OST0000 complete [ 5240.288138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5245.615902] Lustre: server umount lustre-MDT0000 complete [ 5246.980080] LustreError: 281888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818163 with bad export cookie 16935847151139411592 [ 5246.982992] LustreError: 281888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5246.983140] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5247.130228] Lustre: server umount lustre-MDT0001 complete [ 5251.399821] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5252.512420] Key type lgssc unregistered [ 5284.383335] LNet: 285678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5284.386732] LNetError: 285678:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5359.141370] LNet: Removed LNI 192.168.201.121@tcp [ 5359.471134] Key type .llcrypt unregistered [ 5359.473063] Key type ._llcrypt unregistered [ 5365.699731] Key type ._llcrypt registered [ 5365.701565] Key type .llcrypt registered [ 5365.739178] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5372.364682] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5372.866787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5372.890936] alg: No test for adler32 (adler32-zlib) [ 5373.749793] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5373.840680] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5375.431347] Key type lgssc registered [ 5375.841356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5379.117047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5381.761500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5383.719960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5385.793521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5386.449458] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:25:02 (1759818302) [ 5388.704352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5388.723192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5388.730051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.812091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5389.823592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5389.857614] Lustre: lustre-MDT0000: new disk, initializing [ 5389.881878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5389.887595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5391.142230] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5395.345494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5395.372162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5395.409595] Lustre: 288850:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5395.441134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5395.443827] Lustre: Skipped 1 previous similar message [ 5395.486352] Lustre: lustre-MDT0001: new disk, initializing [ 5395.523633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5395.536607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5395.540233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5396.920819] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5399.925891] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5401.131045] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5403.393081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5403.416797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5403.516643] Lustre: lustre-OST0000: new disk, initializing [ 5403.518746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5403.554050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.574779] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5408.793043] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5409.281383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5409.286504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5409.333967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5419.446686] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 02:25:35 (1759818335) [ 5421.023915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5421.027039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.032130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.608484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.610584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.611670] Lustre: Skipped 1 previous similar message [ 5424.619405] Lustre: Skipped 1 previous similar message [ 5426.177077] LustreError: 291488:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5426.207301] Lustre: server umount lustre-MDT0000 complete [ 5427.441070] LustreError: 290594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818344 with bad export cookie 13555268706581116795 [ 5427.444163] LustreError: 290594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5427.444655] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.504346] LustreError: 291689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5427.506232] LustreError: 291689:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5427.576215] Lustre: server umount lustre-MDT0001 complete [ 5439.025408] LustreError: 291889:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5439.028117] LustreError: 291889:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5439.100796] Lustre: server umount lustre-OST0000 complete [ 5454.209779] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 02:26:10 (1759818370) [ 5469.263466] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5470.364634] Key type lgssc unregistered [ 5470.485579] LNet: 294445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.489296] LNetError: 294445:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.498420] LNet: Removed LNI 192.168.201.121@tcp [ 5470.931459] Key type .llcrypt unregistered [ 5470.933497] Key type ._llcrypt unregistered [ 5497.591834] Key type ._llcrypt registered [ 5497.592735] Key type .llcrypt registered [ 5497.630772] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5497.730283] Key type .llcrypt unregistered [ 5497.731348] Key type ._llcrypt unregistered [ 5509.022988] Key type ._llcrypt registered [ 5509.024100] Key type .llcrypt registered [ 5509.060538] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5509.177157] Key type .llcrypt unregistered [ 5509.178161] Key type ._llcrypt unregistered [ 5535.861532] Key type ._llcrypt registered [ 5535.862547] Key type .llcrypt registered [ 5535.897128] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5535.990292] Key type .llcrypt unregistered [ 5535.991287] Key type ._llcrypt unregistered [ 5547.129463] Key type ._llcrypt registered [ 5547.131295] Key type .llcrypt registered [ 5547.171480] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5547.281367] Key type .llcrypt unregistered [ 5547.282907] Key type ._llcrypt unregistered [ 5559.081653] Key type ._llcrypt registered [ 5559.083099] Key type .llcrypt registered [ 5559.120531] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5559.215343] Key type .llcrypt unregistered [ 5559.216464] Key type ._llcrypt unregistered [ 5587.088209] Key type ._llcrypt registered [ 5587.089407] Key type .llcrypt registered [ 5587.123520] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5587.225173] Key type .llcrypt unregistered [ 5587.226170] Key type ._llcrypt unregistered [ 5598.224606] Key type ._llcrypt registered [ 5598.225561] Key type .llcrypt registered [ 5598.257484] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5598.351367] Key type .llcrypt unregistered [ 5598.352993] Key type ._llcrypt unregistered [ 5609.844778] Key type ._llcrypt registered [ 5609.846037] Key type .llcrypt registered [ 5609.881821] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5609.978273] Key type .llcrypt unregistered [ 5609.979762] Key type ._llcrypt unregistered [ 5628.820723] Key type ._llcrypt registered [ 5628.821740] Key type .llcrypt registered [ 5628.856415] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5635.030198] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5635.395721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5635.407120] alg: No test for adler32 (adler32-zlib) [ 5636.258280] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5636.345689] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5637.919129] Key type lgssc registered [ 5638.274418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5641.579577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5644.329691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5646.380971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5648.420345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5648.970501] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5299 sec ========== 02:29:25 (1759818565) [ 5649.577759] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:29:25 (1759818565) === [ 5650.715180] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:29:26 (1759818566) === [ 5663.484128] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5664.560441] Key type lgssc unregistered [ 5664.700329] LNet: 310317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5664.702649] LNetError: 310317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5664.710406] LNet: Removed LNI 192.168.201.121@tcp [ 5665.042278] Key type .llcrypt unregistered [ 5665.043446] Key type ._llcrypt unregistered