[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480710002 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002382] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008502] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011000] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012061] *** VALIDATE selinux *** [ 0.020570] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024758] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] *** VALIDATE tmpfs *** [ 0.029424] *** VALIDATE proc *** [ 0.031229] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.034148] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040719] debug: unmapping init [mem 0xffffffffbb059000-0xffffffffbb060fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043597] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047008] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051282] rcu: Hierarchical SRCU implementation. [ 0.053512] smp: Bringing up secondary CPUs ... [ 0.054604] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.058480] smp: Brought up 1 node, 4 CPUs [ 0.060018] smpboot: Max logical packages: 1 [ 0.061023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.225191] node 0 deferred pages initialised in 160ms [ 0.229010] devtmpfs: initialized [ 0.230258] x86/mm: Memory block size: 128MB [ 0.232934] gcov: version magic: 0x41383552 [ 0.236463] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.240121] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243486] pinctrl core: initialized pinctrl subsystem [ 0.245271] [ 0.245881] ************************************************************* [ 0.248023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250019] ** ** [ 0.253021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255019] ** ** [ 0.258023] ** This means that this kernel is built to expose internal ** [ 0.260019] ** IOMMU data structures, which may compromise security on ** [ 0.262020] ** your system. ** [ 0.264020] ** ** [ 0.266018] ** If you see this message and you are not debugging the ** [ 0.268022] ** kernel, report this immediately to your vendor! ** [ 0.271023] ** ** [ 0.273018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275027] ************************************************************* [ 0.279929] NET: Registered protocol family 16 [ 0.281558] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.284084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.287115] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.290559] cpuidle: using governor menu [ 0.292644] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.295444] PCI: Using configuration type 1 for base access [ 0.297158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.308114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.310027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313133] cryptd: max_cpu_qlen set to 1000 [ 0.317244] ACPI: Added _OSI(Module Device) [ 0.319017] ACPI: Added _OSI(Processor Device) [ 0.320013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326597] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332647] ACPI: Interpreter enabled [ 0.335101] ACPI: PM: (supports S0 S3 S4 S5) [ 0.336017] ACPI: Using IOAPIC for interrupt routing [ 0.338178] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.341324] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.353050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356043] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.365291] acpiphp: Slot [2] registered [ 0.367120] acpiphp: Slot [5] registered [ 0.368132] acpiphp: Slot [6] registered [ 0.370118] acpiphp: Slot [7] registered [ 0.371110] acpiphp: Slot [8] registered [ 0.372129] acpiphp: Slot [9] registered [ 0.373121] acpiphp: Slot [10] registered [ 0.374000] acpiphp: Slot [3] registered [ 0.374000] acpiphp: Slot [4] registered [ 0.374000] acpiphp: Slot [11] registered [ 0.374000] acpiphp: Slot [12] registered [ 0.376111] acpiphp: Slot [13] registered [ 0.377117] acpiphp: Slot [14] registered [ 0.379099] acpiphp: Slot [15] registered [ 0.381098] acpiphp: Slot [16] registered [ 0.382121] acpiphp: Slot [17] registered [ 0.384129] acpiphp: Slot [18] registered [ 0.385134] acpiphp: Slot [19] registered [ 0.387144] acpiphp: Slot [20] registered [ 0.389167] acpiphp: Slot [21] registered [ 0.391144] acpiphp: Slot [22] registered [ 0.392128] acpiphp: Slot [23] registered [ 0.394164] acpiphp: Slot [24] registered [ 0.395150] acpiphp: Slot [25] registered [ 0.397126] acpiphp: Slot [26] registered [ 0.398155] acpiphp: Slot [27] registered [ 0.400128] acpiphp: Slot [28] registered [ 0.401133] acpiphp: Slot [29] registered [ 0.403109] acpiphp: Slot [30] registered [ 0.404148] acpiphp: Slot [31] registered [ 0.406083] PCI host bridge to bus 0000:00 [ 0.407024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.409034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.416034] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.418035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423958] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.427317] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.436014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.441061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.444020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.447014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.451579] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.454733] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.457046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459823] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.465014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.477946] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.482797] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.488858] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.518025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.524023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.538000] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.546805] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.550000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.568023] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.576477] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.583019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.589020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.602019] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.609000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.616023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.621023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635020] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.647484] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.653016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.659016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.674022] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.684981] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.693016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.714018] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.721000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.722496] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.724366] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.727397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734104] iommu: Default domain type: Passthrough [ 0.736468] SCSI subsystem initialized [ 0.737122] ACPI: bus type USB registered [ 0.739110] usbcore: registered new interface driver usbfs [ 0.741078] usbcore: registered new interface driver hub [ 0.743078] usbcore: registered new device driver usb [ 0.745182] pps_core: LinuxPPS API ver. 1 registered [ 0.746012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.750069] PTP clock support registered [ 0.752069] EDAC MC: Ver: 3.0.0 [ 0.754191] PCI: Using ACPI for IRQ routing [ 0.755840] NetLabel: Initializing [ 0.758010] NetLabel: domain hash size = 128 [ 0.759010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761063] NetLabel: unlabeled traffic allowed by default [ 0.764119] vgaarb: loaded [ 0.765272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.767015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.775316] clocksource: Switched to clocksource kvm-clock [ 0.880770] VFS: Disk quotas dquot_6.6.0 [ 0.882560] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.884842] *** VALIDATE ramfs *** [ 0.886190] *** VALIDATE hugetlbfs *** [ 0.887797] pnp: PnP ACPI init [ 0.890162] pnp: PnP ACPI: found 6 devices [ 0.907465] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.911094] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.913365] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.915644] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.918465] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.920841] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.923541] NET: Registered protocol family 2 [ 0.925721] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930900] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.934702] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.942954] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.946544] TCP: Hash tables configured (established 65536 bind 65536) [ 0.949382] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.952289] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955357] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.958250] NET: Registered protocol family 1 [ 0.960973] RPC: Registered named UNIX socket transport module. [ 0.963206] RPC: Registered udp transport module. [ 0.964970] RPC: Registered tcp transport module. [ 0.966764] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969086] NET: Registered protocol family 44 [ 0.970961] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.973222] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.975656] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978199] PCI: CLS 0 bytes, default 64 [ 0.980725] Unpacking initramfs... [ 2.382142] debug: unmapping init [mem 0xffff8e77fcc54000-0xffff8e77fffbffff] [ 2.385724] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388188] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.391232] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.925569] Initialise system trusted keyrings [ 2.927451] Key type blacklist registered [ 2.929451] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.939561] zbud: loaded [ 2.942599] *** VALIDATE nfs *** [ 2.943956] *** VALIDATE nfs4 *** [ 2.945704] pstore: using deflate compression [ 2.948826] Platform Keyring initialized [ 3.060294] NET: Registered protocol family 38 [ 3.061610] Key type asymmetric registered [ 3.062654] Asymmetric key parser 'x509' registered [ 3.063941] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.066056] io scheduler mq-deadline registered [ 3.068796] io scheduler kyber registered [ 3.070376] io scheduler bfq registered [ 3.072316] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.075241] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.078368] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.081166] ACPI: Power Button [PWRF] [ 3.170755] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.262469] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.441910] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.535484] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.720224] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.749055] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.782984] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.787624] Non-volatile memory driver v1.3 [ 3.789540] Linux agpgart interface v0.103 [ 3.823773] virtio_blk virtio1: [vda] 133640 512-byte logical blocks (68.4 MB/65.3 MiB) [ 3.827137] vda: detected capacity change from 0 to 68423680 [ 3.844745] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.847783] vdb: detected capacity change from 0 to 1073741824 [ 3.865039] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.868570] vdc: detected capacity change from 0 to 2621440000 [ 3.886070] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.889083] vdd: detected capacity change from 0 to 2621440000 [ 3.906560] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.909948] vde: detected capacity change from 0 to 4294967296 [ 3.926881] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.929640] vdf: detected capacity change from 0 to 4294967296 [ 3.937281] libphy: Fixed MDIO Bus: probed [ 3.942933] usbcore: registered new interface driver usbserial_generic [ 3.945954] usbserial: USB Serial support registered for generic [ 3.948881] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.953599] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.955592] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.958355] mousedev: PS/2 mouse device common for all mice [ 3.961515] rtc_cmos 00:05: RTC can wake from S4 [ 3.965496] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.965760] rtc_cmos 00:05: registered as rtc0 [ 3.970317] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.971322] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.973764] intel_pstate: CPU model not supported [ 3.977764] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.983914] hid: raw HID events driver (C) Jiri Kosina [ 3.986119] usbcore: registered new interface driver usbhid [ 3.988050] usbhid: USB HID core driver [ 3.989632] drop_monitor: Initializing network drop monitor service [ 3.992204] Initializing XFRM netlink socket [ 3.994223] NET: Registered protocol family 10 [ 3.998488] Segment Routing with IPv6 [ 4.000577] NET: Registered protocol family 17 [ 4.002521] mpls_gso: MPLS GSO support [ 4.007706] RAS: Correctable Errors collector initialized. [ 4.010177] AVX version of gcm_enc/dec engaged. [ 4.011973] AES CTR mode by8 optimization enabled [ 4.091729] sched_clock: Marking stable (4091707028, 0)->(5056640150, -964933122) [ 4.094850] registered taskstats version 1 [ 4.096541] Loading compiled-in X.509 certificates [ 4.101393] zswap: loaded using pool lzo/zbud [ 4.125605] Key type big_key registered [ 4.137041] Key type encrypted registered [ 4.138673] ima: No TPM chip found, activating TPM-bypass! [ 4.140643] ima: Allocated hash algorithm: sha1 [ 4.142562] ima: No architecture policies found [ 4.144466] evm: Initialising EVM extended attributes: [ 4.146556] evm: security.selinux [ 4.147796] evm: security.ima [ 4.148970] evm: security.capability [ 4.150325] evm: HMAC attrs: 0x1 [ 4.152779] rtc_cmos 00:05: setting system clock to 2025-11-14 20:49:41 UTC (1763153381) [ 4.159891] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 4.163509] debug: unmapping init [mem 0xffffffffbad82000-0xffffffffbb058fff] [ 4.176133] Write protecting the kernel read-only data: 28672k [ 4.179769] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 4.182317] debug: unmapping init [mem 0xffffffffb9d14000-0xffffffffb9dfffff] [ 4.217188] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.226801] systemd[1]: Detected virtualization kvm. [ 4.228856] systemd[1]: Detected architecture x86-64. [ 4.230823] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.257122] systemd[1]: No hostname configured. [ 4.259027] systemd[1]: Set hostname to . [ 4.261314] random: systemd: uninitialized urandom read (16 bytes read) [ 4.263624] systemd[1]: Initializing machine ID from random generator. [ 4.291464] random: ln: uninitialized urandom read (6 bytes read) [ 4.388539] random: systemd: uninitialized urandom read (16 bytes read) [ 4.391567] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.395774] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.400993] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.947817] device-mapper: uevent: version 1.0.3 [ 4.950157] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.608331] virtio_net virtio0 ens2: renamed from eth0 [ 5.622248] random: fast init done [ 5.685373] scsi host0: ata_piix [ 5.700942] scsi host1: ata_piix [ 5.703945] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.708368] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.825494] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.658833] random: crng init done [ 10.660055] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.945265] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.051151] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.297103] SELinux: Disabled at runtime. [ 12.358063] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.367092] systemd[1]: Detected virtualization kvm. [ 12.368961] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.850691] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.853870] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.858352] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.865279] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.869573] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.876271] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.880162] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 13.015175] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.319311] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.632675] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.681862] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.812188] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.831612] EDAC sbridge: Ver: 1.1.2 [ 15.667262] Key type dns_resolver registered [ 15.974764] NFS: Registering the id_resolver key type [ 15.976905] Key type id_resolver registered [ 15.978663] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg124-server login: [ 45.953198] libcfs: loading out-of-tree module taints kernel. [ 45.978924] Key type ._llcrypt registered [ 45.980360] Key type .llcrypt registered [ 46.039201] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 57.706964] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 58.789252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.797597] alg: No test for adler32 (adler32-zlib) [ 60.078620] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 60.613689] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 62.303188] Key type lgssc registered [ 63.540279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.961355] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.871338] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.652000] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.055711] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.990668] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.137607] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 112.031371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.071265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 112.085591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 113.296158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 113.322460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 113.392251] Lustre: lustre-MDT0000: new disk, initializing [ 113.471775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.491157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 115.813130] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.236599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.275749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 125.337470] Lustre: 6489:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 125.368651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 125.375387] Lustre: Skipped 1 previous similar message [ 125.432789] Lustre: lustre-MDT0001: new disk, initializing [ 125.486749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 125.525268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 125.532232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 128.088506] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.646108] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 137.862649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 137.906099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 138.061192] Lustre: lustre-OST0000: new disk, initializing [ 138.063551] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 138.066106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 138.116931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.232414] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.338154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.346079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 143.377934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 149.823976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.869207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 149.940919] Lustre: lustre-OST0001: new disk, initializing [ 149.948576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 150.002756] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.620523] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.723985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 157.732736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 157.780469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 162.820307] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 168.410076] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 177.814395] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing check_logdir /tmp/testlogs/ [ 180.580695] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing yml_node [ 183.842973] Lustre: DEBUG MARKER: Client: 2.16.61.2 [ 185.865868] Lustre: DEBUG MARKER: MDS: 2.16.61.2 [ 187.796110] Lustre: DEBUG MARKER: OSS: 2.16.61.2 [ 188.985313] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Nov 14 15:52:44 EST 2025 [ 200.461573] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 201.398569] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 217.570351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 217.573832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 217.582833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.105560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.106063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.109984] Lustre: Skipped 2 previous similar messages [ 219.120112] Lustre: Skipped 2 previous similar messages [ 220.615068] LustreError: 12197:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 220.651913] Lustre: server umount lustre-MDT0000 complete [ 223.518850] LustreError: 6481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153600 with bad export cookie 11275377500355720832 [ 223.519786] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 223.528543] LustreError: 6481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 224.226145] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.248788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 224.251566] Lustre: Skipped 1 previous similar message [ 228.257066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 228.270656] Lustre: Skipped 1 previous similar message [ 229.757619] LustreError: 12400:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 229.762820] LustreError: 12400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 229.864626] Lustre: server umount lustre-MDT0001 complete [ 233.666179] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 233.678265] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 233.771677] Lustre: server umount lustre-OST0000 complete [ 237.299297] LustreError: 12802:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 237.315848] LustreError: 12802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 237.438746] Lustre: server umount lustre-OST0001 complete [ 243.755730] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 250.462613] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 258.404983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 263.889147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 269.624145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 275.263922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 280.505801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 280.571858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.821464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 280.869903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 280.934977] Lustre: lustre-MDT0000: new disk, initializing [ 281.027570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.051647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 284.094597] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 292.098839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 292.138587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 292.209513] Lustre: 14985:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 292.279945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 292.282877] Lustre: Skipped 1 previous similar message [ 292.338159] Lustre: lustre-MDT0001: new disk, initializing [ 292.419949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.462752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 292.471772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 295.450442] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 300.928568] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 303.481137] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 308.309719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.352351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 308.544617] Lustre: lustre-OST0000: new disk, initializing [ 308.549268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 308.605778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 310.477986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 310.485253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 310.573983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 312.470454] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 317.842207] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 320.910710] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 321.097903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 324.285936] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 324.415693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 326.113947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 326.117955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.122800] LustreError: Skipped 1 previous similar message [ 326.127318] Lustre: Skipped 1 previous similar message [ 326.128635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 326.154799] Lustre: Skipped 1 previous similar message [ 331.767119] LustreError: 17922:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 331.779420] LustreError: 17922:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 331.900794] Lustre: server umount lustre-OST0000 complete [ 336.352496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.377692] Lustre: Skipped 2 previous similar messages [ 340.779206] LustreError: 18125:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 340.789518] LustreError: 18125:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 340.879603] Lustre: server umount lustre-MDT0000 complete [ 341.475120] LustreError: 14991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.504877] LustreError: 14991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 343.930285] LustreError: 14977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153721 with bad export cookie 11275377500355722365 [ 343.934765] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.936991] LustreError: 14977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 344.176295] Lustre: server umount lustre-MDT0001 complete [ 348.529346] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:55:24 (1763153724) [ 359.682416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.948076] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.011946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.613892] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 365.024781] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.155187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 372.550581] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 376.433453] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 378.962467] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 380.309151] Lustre: Failing over lustre-MDT0000 [ 380.413744] LustreError: 20805:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 380.417195] LustreError: 20805:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 380.456801] Lustre: server umount lustre-MDT0000 complete [ 384.996311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 385.000095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.004218] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.016081] Lustre: Skipped 1 previous similar message [ 390.113803] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.127676] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 396.046337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.124444] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.717985] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 401.377867] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 401.382215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 401.398356] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 403.125761] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 406.498580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 406.499962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.502104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 406.502111] Lustre: Skipped 4 previous similar messages [ 406.512214] Lustre: Skipped 1 previous similar message [ 411.927192] Lustre: server umount lustre-MDT0000 complete [ 413.664482] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.675481] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 414.503511] LustreError: 21784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153791 with bad export cookie 11275377500355723779 [ 414.507057] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 414.512795] LustreError: 21784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 414.604486] LustreError: 21985:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 414.612872] LustreError: 21985:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 414.721188] Lustre: server umount lustre-MDT0001 complete [ 420.888956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.507541] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 431.192978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 431.496542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.499832] Lustre: Skipped 3 previous similar messages [ 434.079846] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 437.640058] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 439.948553] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 441.357401] Lustre: Failing over lustre-MDT0000 [ 441.832315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 441.837509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.844562] LustreError: 22461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.854676] LustreError: 22461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 443.602558] Lustre: server umount lustre-MDT0000 complete [ 459.916437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.035659] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.248379] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 465.380552] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 465.380926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 465.388187] Lustre: Skipped 1 previous similar message [ 465.405150] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 469.011546] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 475.617556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.622634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.626517] Lustre: Skipped 4 previous similar messages [ 478.619421] LustreError: 24781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 478.623917] LustreError: 24781:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 478.800412] Lustre: server umount lustre-MDT0000 complete [ 481.765228] LustreError: 23173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.797377] LustreError: 23173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 481.928576] LustreError: 22440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153859 with bad export cookie 11275377500355724710 [ 481.943518] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.180022] Lustre: server umount lustre-MDT0001 complete [ 489.655427] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:57:45 (1763153865) [ 499.409060] hrtimer: interrupt took 4374985 ns [ 503.644854] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 511.302152] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 517.433160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 521.983866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 526.153351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 531.117323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 538.298828] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 546.880287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 546.930305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.116696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 547.151605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 547.207171] Lustre: lustre-MDT0000: new disk, initializing [ 547.274468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 550.236318] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 558.590817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 558.653926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 558.723274] Lustre: 28450:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 558.944218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 561.623513] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 567.302886] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 569.914281] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 575.212517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 575.270539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 575.587386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.590433] Lustre: Skipped 3 previous similar messages [ 576.941579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 576.976689] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 577.062931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 577.083907] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 579.788828] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 586.573907] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 595.693133] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 612.835740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 612.836054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.839844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 612.839855] Lustre: Skipped 2 previous similar messages [ 612.851949] Lustre: Skipped 4 previous similar messages [ 617.143623] LustreError: 31110:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 617.146457] LustreError: 31110:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 617.286889] Lustre: server umount lustre-OST0000 complete [ 626.715085] Lustre: server umount lustre-MDT0000 complete [ 628.196283] LustreError: 28455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.221132] LustreError: 28455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 629.792388] LustreError: 28442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154007 with bad export cookie 11275377500355725102 [ 629.799777] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.808870] LustreError: 28442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 630.217189] Lustre: server umount lustre-MDT0001 complete [ 638.358036] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 640.316662] Key type lgssc unregistered [ 640.558995] LNet: 32044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.577572] LNetError: 32044:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.589744] LNet: Removed LNI 192.168.201.124@tcp [ 641.190158] Key type .llcrypt unregistered [ 641.192072] Key type ._llcrypt unregistered [ 659.237972] Key type ._llcrypt registered [ 659.239629] Key type .llcrypt registered [ 659.341181] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 660.233181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 660.247563] alg: No test for adler32 (adler32-zlib) [ 661.341938] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 661.542602] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 663.177121] Key type lgssc registered [ 664.125876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 669.366114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 669.374609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.635050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.073729] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 675.808670] LustreError: 33421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.821676] LustreError: 33421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 679.513381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 679.742565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 682.401918] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 687.954204] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 691.943086] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 697.938048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 698.226956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.016112] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 705.398078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 705.711321] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 731.400256] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:01:47 (1763154107) [ 732.896432] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 734.140081] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 739.203115] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:01:55 (1763154115) [ 746.069689] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:02:01 (1763154121) [ 746.948229] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 753.808550] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 753.811758] Lustre: Skipped 2 previous similar messages [ 755.383066] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:02:11 (1763154131) [ 758.323989] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 758.330885] Lustre: Skipped 3 previous similar messages [ 771.924589] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 771.934087] Lustre: Skipped 6 previous similar messages [ 774.117930] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:02:29 (1763154149) [ 776.286798] Lustre: *** cfs_fail_loc=131b, val=2*** [ 779.414607] Lustre: *** cfs_fail_loc=131b, val=2*** [ 787.065892] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:02:42 (1763154162) [ 789.024859] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 789.027859] Lustre: Skipped 3 previous similar messages [ 800.473469] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:02:56 (1763154176) [ 813.024650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 813.031645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.061173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.056284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.060379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.061759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.077163] Lustre: Skipped 1 previous similar message [ 819.171074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.177375] Lustre: Skipped 1 previous similar message [ 824.287288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 824.301635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 824.314521] Lustre: Skipped 1 previous similar message [ 824.349153] LustreError: 40012:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 824.456155] Lustre: server umount lustre-OST0000 complete [ 828.383942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 828.389349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.398896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.328932] LustreError: 40214:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 833.332560] LustreError: 40214:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 833.374304] Lustre: server umount lustre-MDT0000 complete [ 834.529905] LustreError: 34133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.563370] LustreError: 34133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 836.658233] LustreError: 36685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154214 with bad export cookie 14260559076150411528 [ 836.659669] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 836.671959] LustreError: 36685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 836.842621] LustreError: 40415:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 836.851955] LustreError: 40415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 837.062627] Lustre: server umount lustre-MDT0001 complete [ 845.519452] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 847.838774] Key type lgssc unregistered [ 848.100396] LNet: 40947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.117986] LNetError: 40947:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.137068] LNet: Removed LNI 192.168.201.124@tcp [ 848.893634] Key type .llcrypt unregistered [ 848.895510] Key type ._llcrypt unregistered [ 863.655814] Key type ._llcrypt registered [ 863.658813] Key type .llcrypt registered [ 863.744383] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 876.712520] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 877.399049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.418552] alg: No test for adler32 (adler32-zlib) [ 878.379473] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 878.559621] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 880.207330] Key type lgssc registered [ 881.077363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.493977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 892.855770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 897.568136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 902.443704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 907.272930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 907.334962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 907.345882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.511607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 908.540459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 908.616787] Lustre: lustre-MDT0000: new disk, initializing [ 908.711423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 908.735067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 911.492112] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 920.225586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 920.272286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 920.324911] Lustre: 44071:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 920.358698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 920.365763] Lustre: Skipped 1 previous similar message [ 920.417690] Lustre: lustre-MDT0001: new disk, initializing [ 920.502533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 920.532166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 920.542716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 923.462845] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 929.434329] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 932.753664] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 937.440577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 937.498318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 937.689898] Lustre: lustre-OST0000: new disk, initializing [ 937.694149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 937.754160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.132724] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 946.240355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 946.245537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 946.350952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 948.600302] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 952.152200] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 952.441758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 956.625692] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 956.845685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 962.034152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.041051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 962.068493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.082957] Lustre: Skipped 1 previous similar message [ 962.086072] LustreError: Skipped 1 previous similar message [ 964.612517] LustreError: 47011:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 964.759462] Lustre: server umount lustre-OST0000 complete [ 971.744433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 971.747795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.767320] Lustre: Skipped 1 previous similar message [ 971.780307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.604645] LustreError: 47214:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 974.614308] LustreError: 47214:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 974.696160] Lustre: server umount lustre-MDT0000 complete [ 978.656320] LustreError: 44064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154356 with bad export cookie 14529240459304190157 [ 978.660723] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.681906] LustreError: 44064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 978.828544] LustreError: 47416:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 978.836980] LustreError: 47416:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 979.008907] Lustre: server umount lustre-MDT0001 complete [ 988.830781] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:06:04 (1763154364) [ 995.285492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.815886] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.834766] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 995.881989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.246787] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1000.937744] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.053101] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.215165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1009.511660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.204895] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1019.004932] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1023.288744] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.629086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.019316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.509281] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1042.360983] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.897916] LustreError: 50333:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1048.985485] Lustre: Modifying parameter general.timeout=1 in log params [ 1050.010608] Lustre: Modifying parameter general.timeout=2 in log params [ 1052.680746] Lustre: Modifying parameter general.timeout=5 in log params [ 1052.682984] Lustre: Skipped 2 previous similar messages [ 1057.240287] Lustre: Modifying parameter general.timeout=10 in log params [ 1057.242538] Lustre: Skipped 4 previous similar messages [ 1065.565229] Lustre: Modifying parameter general.timeout=21 in log params [ 1065.567741] Lustre: Skipped 10 previous similar messages [ 1081.568924] Lustre: Modifying parameter general.timeout=42 in log params [ 1081.571801] Lustre: Skipped 20 previous similar messages [ 1113.661055] Lustre: Modifying parameter general.timeout=86 in log params [ 1113.676975] Lustre: Skipped 43 previous similar messages [ 1154.246510] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:08:50 (1763154530) [ 1156.783551] Lustre: 57267:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1158.352736] Lustre: 57365:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1161.654947] Lustre: 57581:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1161.673832] Lustre: 57581:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1178.779840] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1178.783236] Lustre: Skipped 61 previous similar messages [ 1187.288733] Lustre: 59129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1187.294281] Lustre: 59129:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1187.314056] LustreError: 59129:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1190.675760] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:09:26 (1763154566) [ 1201.633916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.636190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.672554] Lustre: Skipped 3 previous similar messages [ 1201.690770] Lustre: Skipped 4 previous similar messages [ 1205.620057] LustreError: 59830:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1205.626911] LustreError: 59830:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1205.718769] Lustre: server umount lustre-MDT0000 complete [ 1206.755236] LustreError: 48129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.770926] LustreError: 48129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1209.475020] LustreError: 48116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154586 with bad export cookie 14529240459304191046 [ 1209.485995] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1209.486223] LustreError: 48116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1209.629341] LustreError: 60031:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1209.662290] LustreError: 60031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1209.800438] Lustre: server umount lustre-MDT0001 complete [ 1223.491621] LustreError: 60232:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1223.496404] LustreError: 60232:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1223.618539] Lustre: server umount lustre-OST0000 complete [ 1230.192813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1233.308162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1236.160788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1239.275105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1244.452711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.641278] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1244.887188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.944101] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1254.856847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.950587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1254.986632] Lustre: 61682:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1258.472424] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1262.611273] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1265.341953] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1272.007644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1272.154953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1277.558606] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1282.261564] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1283.757213] Lustre: 63916:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1289.701314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.709401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.712288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.723871] Lustre: Skipped 1 previous similar message [ 1290.684240] LustreError: 64175:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1290.686952] LustreError: 64175:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1290.775142] Lustre: server umount lustre-OST0000 complete [ 1294.819641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.821674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.833266] Lustre: Skipped 1 previous similar message [ 1294.835467] Lustre: Skipped 1 previous similar message [ 1299.714705] Lustre: server umount lustre-MDT0000 complete [ 1299.936623] LustreError: 62415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.944249] LustreError: 62415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1302.292938] LustreError: 63334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154679 with bad export cookie 14529240459304193951 [ 1302.296548] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.308342] LustreError: 63334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1302.589474] Lustre: server umount lustre-MDT0001 complete [ 1310.561246] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1312.555155] Key type lgssc unregistered [ 1312.801067] LNet: 65109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.808777] LNetError: 65109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.832473] LNet: Removed LNI 192.168.201.124@tcp [ 1314.478543] Key type .llcrypt unregistered [ 1314.480509] Key type ._llcrypt unregistered [ 1334.242898] Key type ._llcrypt registered [ 1334.244458] Key type .llcrypt registered [ 1334.371212] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1335.137972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.218883] alg: No test for adler32 (adler32-zlib) [ 1336.263695] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1336.456203] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1338.108861] Key type lgssc registered [ 1339.083195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.278239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.296161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.788621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.879490] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1357.117184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1357.553555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.989135] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1366.380224] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1369.497761] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1374.917925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1375.358438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.815685] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1384.171093] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1399.785720] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1407.462501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.479757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1407.485659] Lustre: Skipped 2 previous similar messages [ 1407.502060] Lustre: Skipped 1 previous similar message [ 1412.580890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.589538] Lustre: Skipped 3 previous similar messages [ 1412.604683] LustreError: 69225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1412.688475] Lustre: server umount lustre-MDT0000 complete [ 1415.715194] LustreError: 66477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154793 with bad export cookie 15322222765907366420 [ 1415.720833] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.849307] LustreError: 69428:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1415.863448] LustreError: 69428:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1415.983391] Lustre: server umount lustre-MDT0001 complete [ 1429.581628] LustreError: 69630:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1429.585534] LustreError: 69630:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1429.705665] Lustre: server umount lustre-OST0000 complete [ 1436.745958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1440.267890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1443.467085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.809577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1453.967619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.345255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1454.396032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1454.703509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.483812] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1466.863535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1466.988978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1467.036916] Lustre: 71082:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1467.558321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.452849] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1476.668899] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.270346] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1486.962808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1487.165883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1487.561128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.543632] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1497.892752] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1499.493414] Lustre: 73326:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1501.126201] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1509.344905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.348420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.371583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.858069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.373103] LustreError: 73583:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1511.375958] LustreError: 73583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1511.559405] Lustre: server umount lustre-OST0000 complete [ 1518.560878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1518.571477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.576455] Lustre: Skipped 1 previous similar message [ 1518.598938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.601585] Lustre: Skipped 1 previous similar message [ 1521.687484] LustreError: 73784:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1521.696982] LustreError: 73784:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1521.821444] Lustre: server umount lustre-MDT0000 complete [ 1525.216274] LustreError: 72091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.225819] LustreError: 72091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1525.981034] LustreError: 71072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154903 with bad export cookie 15322222765907367253 [ 1525.985816] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.995556] LustreError: 71072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1526.507860] Lustre: server umount lustre-MDT0001 complete [ 1537.036953] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1539.576592] Key type lgssc unregistered [ 1540.009466] LNet: 74518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.018646] LNetError: 74518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.033734] LNet: Removed LNI 192.168.201.124@tcp [ 1540.918201] Key type .llcrypt unregistered [ 1540.920805] Key type ._llcrypt unregistered [ 1560.175916] Key type ._llcrypt registered [ 1560.179021] Key type .llcrypt registered [ 1560.270865] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1560.962495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.010776] alg: No test for adler32 (adler32-zlib) [ 1562.028296] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1562.235047] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1563.895201] Key type lgssc registered [ 1564.820391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1571.313284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1571.337599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.817669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.154898] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1583.077842] LustreError: 75891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.104382] LustreError: 75891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1584.989411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1585.771863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1590.244358] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1596.074545] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1599.544242] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1606.045622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1606.564843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.090042] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1616.285610] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1626.077214] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1627.013452] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1628.160255] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1631.077648] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1631.082660] Lustre: Skipped 2 previous similar messages [ 1635.124215] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1635.133289] Lustre: Skipped 3 previous similar messages [ 1643.550199] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1643.555423] Lustre: Skipped 8 previous similar messages [ 1659.889854] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1659.893026] Lustre: Skipped 19 previous similar messages [ 1692.112270] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1692.115386] Lustre: Skipped 39 previous similar messages [ 1720.856790] Lustre: 83663:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1723.617273] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:18:19 (1763155099) [ 1725.474564] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1727.173659] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:18:22 (1763155102) [ 1730.528996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.534521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.554199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.649050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.651876] Lustre: Skipped 1 previous similar message [ 1736.527229] LustreError: 83959:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1736.650329] Lustre: server umount lustre-OST0000 complete [ 1740.772703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.777066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.779535] Lustre: Skipped 2 previous similar messages [ 1745.892558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.896176] Lustre: Skipped 2 previous similar messages [ 1745.910856] LustreError: 84160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1745.914372] LustreError: 84160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1746.008162] Lustre: server umount lustre-MDT0000 complete [ 1749.451318] LustreError: 75871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155126 with bad export cookie 217533546768166308 [ 1749.459128] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.474456] LustreError: 75871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1749.673610] LustreError: 84362:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1749.684483] LustreError: 84362:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1749.933534] Lustre: server umount lustre-MDT0001 complete [ 1759.689945] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1761.686231] Key type lgssc unregistered [ 1761.892817] LNet: 84893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.903862] LNetError: 84893:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.917836] LNet: Removed LNI 192.168.201.124@tcp [ 1762.574592] Key type .llcrypt unregistered [ 1762.578478] Key type ._llcrypt unregistered [ 1765.973832] Key type ._llcrypt registered [ 1765.982473] Key type .llcrypt registered [ 1766.110483] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1769.193218] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules [ 1770.089730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.099366] alg: No test for adler32 (adler32-zlib) [ 1771.113759] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1771.306786] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1772.975364] Key type lgssc registered [ 1773.763723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1774.704298] LustreError: 86058:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1789.366784] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1796.908915] LustreError: 86058:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1797.539305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.579555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.226355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.552973] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1809.378713] LustreError: 87064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.389264] LustreError: 87064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1810.050163] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:19:45 (1763155185) [ 1813.585581] LustreError: 87096:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1813.597791] LustreError: 87096:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1813.602876] LustreError: 87096:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 1813.734390] LustreError: 87870:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1813.904776] Lustre: server umount lustre-MDT0000 complete [ 1826.497703] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1829.226939] Key type lgssc unregistered [ 1829.555756] LNet: 88554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.577172] LNetError: 88554:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.596997] LNet: Removed LNI 192.168.201.124@tcp [ 1830.304799] Key type .llcrypt unregistered [ 1830.306782] Key type ._llcrypt unregistered [ 1847.418967] Key type ._llcrypt registered [ 1847.420599] Key type .llcrypt registered [ 1847.531450] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1848.280146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.300885] alg: No test for adler32 (adler32-zlib) [ 1849.344978] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1849.584463] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1851.255758] Key type lgssc registered [ 1852.426750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1860.119562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1860.155483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.660156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.739869] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1870.817559] LustreError: 89753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.853618] LustreError: 89753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1874.168593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1874.606347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.955032] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1882.751739] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.077503] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1892.113182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1892.623859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.307959] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1902.094308] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1949.421191] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:22:04 (1763155324) [ 1951.323532] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1953.106103] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:22:08 (1763155328) [ 1957.345087] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.349805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.371074] Lustre: Skipped 1 previous similar message [ 1957.395090] Lustre: Skipped 2 previous similar messages [ 1962.466670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.469870] Lustre: Skipped 1 previous similar message [ 1962.934567] LustreError: 92668:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1963.072506] Lustre: server umount lustre-MDT0000 complete [ 1966.559751] LustreError: 91383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155343 with bad export cookie 17088703413064305137 [ 1966.564546] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.585521] LustreError: 91383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1966.743613] LustreError: 92869:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1966.763482] LustreError: 92869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1966.915709] Lustre: server umount lustre-MDT0001 complete [ 1980.375356] LustreError: 93070:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1980.389352] LustreError: 93070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1980.651455] Lustre: server umount lustre-OST0000 complete [ 1991.925867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.239325] LustreError: 93700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.381437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.621332] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1997.799055] LustreError: 93700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.890466] LustreError: 93700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.094712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2003.838849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.409749] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2011.750992] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2014.461677] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2019.874651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2026.668644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2026.757901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2027.022566] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2027.027585] LustreError: 93689:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2027.045695] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2027.052378] LustreError: 95562:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2027.059804] LustreError: 95562:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2027.064241] LustreError: 95562:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2027.137310] Lustre: server umount lustre-OST0000 complete [ 2027.139888] LustreError: 95562:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2034.744616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2034.870780] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2034.876369] LustreError: 93689:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2034.884732] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2034.889739] LustreError: 96024:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2034.894614] LustreError: 96024:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2034.899277] LustreError: 96024:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2034.949073] Lustre: server umount lustre-OST0000 complete [ 2034.951350] LustreError: 96024:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2038.053062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2043.717613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2043.838690] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2043.841820] Lustre: Found index 0 for lustre-OST0000, updating log [ 2043.845484] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2043.945341] Lustre: lustre-OST0000: new disk, initializing [ 2043.948436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2044.184633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.504180] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2051.287398] Lustre: Failing over lustre-OST0000 [ 2051.326186] LustreError: 97112:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2051.332444] LustreError: 97112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2051.403334] Lustre: server umount lustre-OST0000 complete [ 2054.624463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2054.629081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.636473] Lustre: Skipped 1 previous similar message [ 2054.640370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.720495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.723344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.744877] Lustre: Skipped 1 previous similar message [ 2060.111477] LustreError: 97313:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2060.115938] LustreError: 97313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2060.200251] Lustre: server umount lustre-MDT0000 complete [ 2062.957623] LustreError: 93681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155440 with bad export cookie 17088703413064307888 [ 2062.969323] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.579466] LustreError: 97514:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2073.584625] LustreError: 97514:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2073.751905] Lustre: server umount lustre-MDT0001 complete [ 2082.072548] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:24:17 (1763155457) [ 2096.234583] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2107.193721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.535065] LustreError: 98885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.634160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.571018] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2117.090388] LustreError: 98885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.115551] LustreError: 98885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2120.234575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2125.076250] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2128.135530] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2135.218444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2136.007658] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2141.177088] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2145.203934] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2145.267153] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2145.267474] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2149.373700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2149.434377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.474368] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2149.533821] Lustre: lustre-OST0001: new disk, initializing [ 2149.537200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2149.755280] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2149.758504] Lustre: Skipped 2 previous similar messages [ 2152.738899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2152.753224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2152.896856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2154.363768] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2160.522926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2165.923705] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2176.996381] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.004858] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.052437] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2180.796734] LustreError: 102126:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2180.804197] LustreError: 102126:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2180.874125] Lustre: server umount lustre-MDT0001 complete [ 2182.180103] LustreError: 98880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.486262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2189.487646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2189.589418] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2189.594236] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2189.640919] Lustre: 98872:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2191.630279] LustreError: 102727:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2191.656987] LustreError: 102727:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2191.688228] LustreError: 98876:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2191.817076] LustreError: 102740:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -17 [ 2191.821676] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2191.821676] [ 2194.219950] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2195.943491] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2204.035629] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 16:26:19 (1763155579) [ 2218.112868] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2234.288142] Lustre: 104738:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2253.827171] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2257.273820] Lustre: 105872:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2343.394258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.410316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.424752] Lustre: Skipped 2 previous similar messages [ 2343.427821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.430640] Lustre: Skipped 4 previous similar messages [ 2349.673217] LustreError: 106333:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2349.687862] LustreError: 106333:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2349.798831] Lustre: server umount lustre-MDT0000 complete [ 2353.437541] LustreError: 98865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155730 with bad export cookie 17088703413064308819 [ 2353.445539] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.459886] LustreError: 98865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2353.833474] Lustre: server umount lustre-MDT0001 complete [ 2369.144510] Lustre: server umount lustre-OST0000 complete [ 2371.039142] Lustre: 89065:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155732/real 1763155732] req@ffff8e786f309180 x1848802254606848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2371.056311] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.067337] Lustre: Skipped 3 previous similar messages [ 2372.437519] Lustre: server umount lustre-OST0001 complete [ 2376.982700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2383.280160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.076441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2390.561793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2398.925617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2399.883516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2420.465313] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2433.377550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.396670] Lustre: lustre-MDT0000: reset Object Index mappings [ 2433.887123] LustreError: 109818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.895962] LustreError: 109818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2434.016865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.027428] Lustre: Skipped 1 previous similar message [ 2437.520580] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2444.886798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2444.905903] Lustre: lustre-MDT0001: reset Object Index mappings [ 2450.058639] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2453.289206] Lustre: 110933:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2460.868465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2468.150307] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2470.651229] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 2477.377093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2479.041624] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2479.207238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2479.239594] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2483.589347] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2492.130426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2496.230195] Lustre: 112789:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2547.788821] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:32:03 (1763155923) [ 2576.357698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.362448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.385043] Lustre: Skipped 3 previous similar messages [ 2576.398562] Lustre: Skipped 11 previous similar messages [ 2580.169991] LustreError: 113147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2580.176108] LustreError: 113147:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2580.319595] Lustre: server umount lustre-MDT0000 complete [ 2581.474447] LustreError: 110538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.516850] LustreError: 110538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2584.498417] LustreError: 109799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155961 with bad export cookie 17088703413064355712 [ 2584.505903] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.523760] LustreError: 109799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2590.994870] Lustre: server umount lustre-MDT0001 complete [ 2600.808273] Lustre: server umount lustre-OST0000 complete [ 2610.568912] Lustre: server umount lustre-OST0001 complete [ 2617.064773] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 2626.453399] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2635.804476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2642.267928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2648.119471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2654.352895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.771843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.832417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.016424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2660.059096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2660.115108] Lustre: lustre-MDT0000: new disk, initializing [ 2660.192295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.197669] Lustre: Skipped 3 previous similar messages [ 2660.211600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2663.704344] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2669.635802] Lustre: server umount lustre-MDT0000 complete [ 2672.222575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2678.572621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2678.691359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.069106] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2679.088502] Lustre: 117025:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2679.102504] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2682.790810] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2687.794568] Lustre: server umount lustre-MDT0000 complete [ 2696.938449] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:34:32 (1763156072) [ 2698.710178] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2700.604777] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:34:36 (1763156076) [ 2716.571911] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2727.038272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.931104] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2738.839871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2738.903544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2739.037624] Lustre: 119044:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.159462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2739.162373] Lustre: Skipped 1 previous similar message [ 2739.282117] Lustre: lustre-MDT0001: new disk, initializing [ 2739.463710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2739.483662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2743.801288] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2748.300656] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2754.057102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2754.131066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2754.347395] Lustre: lustre-OST0000: new disk, initializing [ 2754.355576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2756.175447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2756.185646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2756.326778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2759.928197] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2769.354560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.433712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2769.585589] Lustre: lustre-OST0001: new disk, initializing [ 2769.598853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2770.723557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2770.733361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2770.805168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2774.604239] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2783.884250] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2787.111535] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3119.355270] Lustre: 127840:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3119.362272] Lustre: 127840:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3119.367073] Lustre: 127840:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3119.374308] Lustre: 127840:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3119.379465] Lustre: 127840:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3119.383847] Lustre: 127840:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3327.304845] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:45:03 (1763156703) [ 3344.359450] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.363101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.363236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3344.365889] Lustre: Skipped 4 previous similar messages [ 3344.370908] Lustre: Skipped 5 previous similar messages [ 3345.821457] LustreError: 128065:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3345.823731] LustreError: 128065:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3345.926667] Lustre: server umount lustre-MDT0000 complete [ 3347.899264] LustreError: 120592:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156725 with bad export cookie 17088703413064423304 [ 3347.902836] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3347.904598] LustreError: 120592:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3349.476605] LustreError: 122248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.480126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.496491] LustreError: 122248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3349.506528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3349.510177] Lustre: Skipped 2 previous similar messages [ 3357.665768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3357.669688] Lustre: Skipped 4 previous similar messages [ 3362.271127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3362.324849] LustreError: 128267:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3362.327300] LustreError: 128267:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3362.456039] Lustre: server umount lustre-MDT0001 complete [ 3379.167177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3379.225185] Lustre: server umount lustre-OST0000 complete [ 3387.292278] Lustre: server umount lustre-OST0001 complete [ 3390.392567] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3396.096498] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3401.814414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3405.401178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3408.693084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.208381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3415.875454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3415.918784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.098512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3416.130539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3416.175780] Lustre: lustre-MDT0000: new disk, initializing [ 3416.231068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.236497] Lustre: Skipped 5 previous similar messages [ 3416.250266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3418.366521] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3424.360525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3424.403540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3424.453054] Lustre: 130857:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3424.489840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3424.492594] Lustre: Skipped 1 previous similar message [ 3424.536027] Lustre: lustre-MDT0001: new disk, initializing [ 3424.607156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3424.611214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3426.527242] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3430.571610] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3432.308903] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3435.640332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.677482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3435.824670] Lustre: lustre-OST0000: new disk, initializing [ 3435.827860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3437.706339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3437.713921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3437.761457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3438.753450] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3442.936735] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.556237] Lustre: lustre-MDD0000: changelog on [ 3449.009294] Lustre: lustre-MDD0001: changelog on [ 4247.188946] Lustre: lustre-MDD0001: changelog off [ 4247.992278] Lustre: 133212:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4248.431697] Lustre: lustre-MDD0000: changelog off [ 4252.130537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4252.135337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.142220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4252.144866] Lustre: Skipped 3 previous similar messages [ 4256.752474] LustreError: 140479:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4256.756023] LustreError: 140479:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4256.811192] Lustre: server umount lustre-OST0000 complete [ 4259.295934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4259.299041] LustreError: Skipped 1 previous similar message [ 4259.300903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.305932] Lustre: Skipped 1 previous similar message [ 4259.308404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4262.368906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.517649] LustreError: 140680:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4264.520765] LustreError: 140680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4264.587308] Lustre: server umount lustre-MDT0000 complete [ 4265.997438] LustreError: 130848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157643 with bad export cookie 17088703413065237264 [ 4266.000287] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.002377] LustreError: 130848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4266.130437] Lustre: server umount lustre-MDT0001 complete [ 4270.362637] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4271.563727] Key type lgssc unregistered [ 4271.717564] LNet: 141412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4271.721753] LNetError: 141412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4271.736428] LNet: Removed LNI 192.168.201.124@tcp [ 4272.070138] Key type .llcrypt unregistered [ 4272.071687] Key type ._llcrypt unregistered [ 4279.814923] Key type ._llcrypt registered [ 4279.816554] Key type .llcrypt registered [ 4279.856021] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4287.841793] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4288.247562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4288.294443] alg: No test for adler32 (adler32-zlib) [ 4289.170446] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4289.270856] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4290.863214] Key type lgssc registered [ 4291.307663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.921776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4297.592358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4300.085664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4302.685764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4305.485439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4305.508082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4305.516094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.618328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4306.638598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4306.681082] Lustre: lustre-MDT0000: new disk, initializing [ 4306.720664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.729938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4308.336323] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4313.105460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4313.132476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4313.172837] Lustre: 144572:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4313.197981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4313.200584] Lustre: Skipped 1 previous similar message [ 4313.235480] Lustre: lustre-MDT0001: new disk, initializing [ 4313.272478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4313.287511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4313.291367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4314.830764] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4318.350049] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4319.743358] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4322.372221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4322.400307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.536222] Lustre: lustre-OST0000: new disk, initializing [ 4322.538991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4322.583285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.802169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4322.806849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4322.862388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4324.871784] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4328.384813] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4330.784651] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_module obdecho/obdecho [ 4331.432987] LustreError: 147200:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4331.436293] LustreError: 147200:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4332.048637] LustreError: 147296:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4338.148124] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.149867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.153228] Lustre: Skipped 1 previous similar message [ 4341.760367] LustreError: 147635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4341.793406] Lustre: server umount lustre-MDT0000 complete [ 4343.329685] LustreError: 144565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157720 with bad export cookie 824833008589646204 [ 4343.334746] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.334956] LustreError: 144565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4343.407635] LustreError: 147837:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4343.409856] LustreError: 147837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4343.472852] Lustre: server umount lustre-MDT0001 complete [ 4355.185933] LustreError: 148037:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4355.188522] LustreError: 148037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4355.237745] Lustre: server umount lustre-OST0000 complete [ 4362.940372] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4366.747841] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4370.718685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4373.248294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4375.722581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4378.425079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4381.227823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4381.254101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.358255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4381.376817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4381.419395] Lustre: lustre-MDT0000: new disk, initializing [ 4381.449579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4381.460346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4383.068801] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4388.013518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4388.044384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4388.090379] Lustre: 150979:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4388.123182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4388.126423] Lustre: Skipped 1 previous similar message [ 4388.166423] Lustre: lustre-MDT0001: new disk, initializing [ 4388.211962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4388.227899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4388.232439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4389.792920] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4393.247855] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4394.635705] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.129738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4397.155487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4397.253598] Lustre: lustre-OST0000: new disk, initializing [ 4397.256668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4397.282787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.387680] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4402.757859] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4403.201157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4403.206067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4403.245902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4404.304636] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4404.393950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4405.962756] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4406.054649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4408.289388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4408.290158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.293165] LustreError: Skipped 1 previous similar message [ 4408.293802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.297555] Lustre: Skipped 2 previous similar messages [ 4408.302221] Lustre: Skipped 3 previous similar messages [ 4412.846072] LustreError: 153920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4412.848507] LustreError: 153920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4412.892340] Lustre: server umount lustre-OST0000 complete [ 4418.528875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.529523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.533876] Lustre: Skipped 1 previous similar message [ 4418.538402] Lustre: Skipped 1 previous similar message [ 4420.351221] LustreError: 154121:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4420.353528] LustreError: 154121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4420.379434] Lustre: server umount lustre-MDT0000 complete [ 4421.686474] LustreError: 151762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157799 with bad export cookie 824833008589647149 [ 4421.690361] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4421.690501] LustreError: 151762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4421.801287] Lustre: server umount lustre-MDT0001 complete [ 4423.143830] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:03:19 (1763157799) [ 4423.781980] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4424.483930] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:03:20 (1763157800) [ 4427.323845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.477507] LustreError: 154994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.485757] LustreError: 154994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4427.505339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.909614] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4432.104881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4433.680863] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4435.826846] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4437.350931] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4439.863758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4442.148099] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4444.111070] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4457.442137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.443288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.448073] Lustre: Skipped 2 previous similar messages [ 4457.452561] Lustre: Skipped 2 previous similar messages [ 4462.560668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.563595] Lustre: Skipped 3 previous similar messages [ 4462.918391] LustreError: 157262:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4462.920843] LustreError: 157262:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4462.955552] Lustre: server umount lustre-MDT0000 complete [ 4464.326568] LustreError: 154974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157841 with bad export cookie 824833008589648031 [ 4464.331385] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4464.331380] LustreError: 154974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4464.670601] LustreError: 154988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.679867] LustreError: 154988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.681545] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.727873] LustreError: 154989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.520156] Lustre: server umount lustre-MDT0001 complete [ 4472.961035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4477.551166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4477.661546] LustreError: 158523:0:(llog.c:1616:llog_backup()) MGC192.168.201.124@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4477.666776] Lustre: 158523:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.124@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4494.303901] LustreError: 158531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.308980] LustreError: 158531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4504.543745] LustreError: 158549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4504.550030] LustreError: 158549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4520.989258] LustreError: 158529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.993933] LustreError: 158529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4538.463227] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4538.465218] Lustre: 158523:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4555.231926] LustreError: 158529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.237136] LustreError: 158529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 4563.999230] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4564.001034] Lustre: 154985:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4579.301910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4579.303706] Lustre: Skipped 2 previous similar messages [ 4580.547298] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4583.288898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.415263] LustreError: 154985:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4584.420536] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4585.375148] LustreError: 159251:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 4585.379109] LustreError: 159251:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000043fe420c x1848804812879360/t0(0) o250->MGC192.168.201.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1763157962 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.386624] LustreError: 159251:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 4585.440561] LustreError: 142785:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cda49a28 x1848804812883840/t0(0) o250->MGC192.168.201.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.501550] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (not set up) [ 4585.505029] Lustre: Skipped 4 previous similar messages [ 4587.037822] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4589.038692] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4590.564848] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4591.736371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4593.029166] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4593.678169] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4599.530292] LustreError: 160006:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4599.532957] LustreError: 160006:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4599.561644] Lustre: server umount lustre-MDT0000 complete [ 4600.767589] LustreError: 154973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157978 with bad export cookie 824833008589654492 [ 4600.770915] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.771425] LustreError: 154973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4600.801456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.805241] Lustre: Skipped 1 previous similar message [ 4600.904602] Lustre: server umount lustre-MDT0001 complete [ 4608.284501] Lustre: server umount lustre-OST0000 complete [ 4616.667350] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4620.183858] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4623.766329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4626.482153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.932482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4631.393735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4633.884506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4633.905150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4633.983500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4633.994194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4634.027868] Lustre: lustre-MDT0000: new disk, initializing [ 4634.053655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4635.429934] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4639.769164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.793612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4639.826684] Lustre: 163350:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4639.938097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4641.364749] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4644.612974] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4645.987601] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4648.241408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4648.261854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4648.375843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4648.378813] Lustre: Skipped 3 previous similar messages [ 4650.039079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4650.076083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4650.204679] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4653.348473] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4654.780105] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4654.855277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4656.244143] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4656.322454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4660.194117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.194866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4660.198349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.201085] LustreError: Skipped 1 previous similar message [ 4660.202871] Lustre: Skipped 1 previous similar message [ 4663.022082] LustreError: 166287:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4663.024711] LustreError: 166287:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4663.062396] Lustre: server umount lustre-OST0000 complete [ 4670.481585] Lustre: server umount lustre-MDT0000 complete [ 4671.736116] LustreError: 163341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158049 with bad export cookie 824833008589671271 [ 4671.737993] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4671.739417] LustreError: 163341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4671.847087] Lustre: server umount lustre-MDT0001 complete [ 4675.068659] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:07:31 (1763158051) [ 4677.317793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.466475] LustreError: 167359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.473624] LustreError: 167359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 4678.810890] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4681.696676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4683.195166] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4685.073335] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4686.396534] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4688.802138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4690.714855] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4692.470762] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4712.290345] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:08:08 (1763158088) [ 4713.441819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.442126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4713.442825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4713.442828] Lustre: Skipped 5 previous similar messages [ 4713.446490] Lustre: Skipped 4 previous similar messages [ 4719.377562] Lustre: server umount lustre-OST0000 complete [ 4722.655899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4726.881391] Lustre: server umount lustre-MDT0000 complete [ 4728.202355] LustreError: 167339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158105 with bad export cookie 824833008589672146 [ 4728.206481] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4728.206923] LustreError: 167339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4728.280709] LustreError: 170226:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4728.283778] LustreError: 170226:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 4728.342847] Lustre: server umount lustre-MDT0001 complete [ 4732.182664] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4733.177863] Key type lgssc unregistered [ 4733.303324] LNet: 170758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4733.306059] LNetError: 170758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4733.312258] LNet: Removed LNI 192.168.201.124@tcp [ 4733.615108] Key type .llcrypt unregistered [ 4733.616351] Key type ._llcrypt unregistered [ 4741.086555] Key type ._llcrypt registered [ 4741.087391] Key type .llcrypt registered [ 4741.122723] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4741.417676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4741.446165] alg: No test for adler32 (adler32-zlib) [ 4742.286743] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4742.358799] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4743.935167] Key type lgssc registered [ 4744.304471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4747.050153] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4747.056892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4762.527339] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4767.647136] LustreError: 171958:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 4772.767468] LustreError: 171958:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4772.772791] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4772.775458] LustreError: MGC192.168.201.124@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4772.779682] LustreError: 171958:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4772.782324] LustreError: 171958:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 4772.785653] LustreError: 171958:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4772.788695] LustreError: 171958:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4772.810099] Lustre: server umount lustre-OST0000 complete [ 4772.811434] LustreError: 171958:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4775.860023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.009447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.290845] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4781.176953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4781.295094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4782.584931] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4784.340412] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4785.542080] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4787.792184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4787.880088] LustreError: 174049:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4787.883696] Lustre: 174049:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.124@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4787.934700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.908373] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4791.760729] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4796.387612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.389449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4796.391264] Lustre: Skipped 2 previous similar messages [ 4796.396370] Lustre: Skipped 2 previous similar messages [ 4798.716679] LustreError: 174697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4798.719482] LustreError: 174697:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4798.748690] Lustre: server umount lustre-MDT0000 complete [ 4800.044546] LustreError: 173178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158177 with bad export cookie 5975226564624040835 [ 4800.047084] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4800.049070] LustreError: 173178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4800.109340] LustreError: 174901:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4800.112151] LustreError: 174901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4800.153119] Lustre: server umount lustre-MDT0001 complete [ 4811.632301] LustreError: 175101:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4811.634136] LustreError: 175101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4811.674409] Lustre: server umount lustre-OST0000 complete [ 4814.791185] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4818.052985] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4821.221705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.426532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4825.573116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.766199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4829.972183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4829.993047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4830.067887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4830.078977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4830.112062] Lustre: lustre-MDT0000: new disk, initializing [ 4830.135647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4830.141571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4831.316984] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4835.306449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4835.324723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4835.356621] Lustre: 177434:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.377834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4835.379479] Lustre: Skipped 1 previous similar message [ 4835.410564] Lustre: lustre-MDT0001: new disk, initializing [ 4835.440871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4835.453236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4835.457573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4836.675269] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4839.841981] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4841.212856] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4843.683652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4843.706039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4843.809642] Lustre: lustre-OST0000: new disk, initializing [ 4843.812953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4843.844137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.903789] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4849.015375] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4849.653898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4849.658085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4849.689269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.380883] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4850.446874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4851.744643] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4851.809523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4854.754686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4854.757251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.759180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4854.764505] Lustre: Skipped 1 previous similar message [ 4858.477649] LustreError: 180374:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4858.479374] LustreError: 180374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4858.515375] Lustre: server umount lustre-OST0000 complete [ 4859.872352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4859.872820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4859.875761] Lustre: Skipped 1 previous similar message [ 4859.877523] Lustre: Skipped 1 previous similar message [ 4864.992268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4864.994678] Lustre: Skipped 2 previous similar messages [ 4865.822610] Lustre: server umount lustre-MDT0000 complete [ 4867.136413] LustreError: 178215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158244 with bad export cookie 5975226564624041521 [ 4867.141677] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.142818] LustreError: 178215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4867.213807] LustreError: 180776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4867.217540] LustreError: 180776:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4867.274528] Lustre: server umount lustre-MDT0001 complete [ 4870.527453] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:10:47 (1763158247) [ 4876.823811] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4877.891807] Key type lgssc unregistered [ 4878.022423] LNet: 181958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4878.024730] LNetError: 181958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4878.033331] LNet: Removed LNI 192.168.201.124@tcp [ 4878.336852] Key type .llcrypt unregistered [ 4878.338469] Key type ._llcrypt unregistered [ 4885.760251] Key type ._llcrypt registered [ 4885.761174] Key type .llcrypt registered [ 4885.792939] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4886.084110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4886.140632] alg: No test for adler32 (adler32-zlib) [ 4886.985035] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4887.056651] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4888.631166] Key type lgssc registered [ 4888.974599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4891.652770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4891.659146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.806079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.112817] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4897.189312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4897.306552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4898.634471] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4900.432534] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4901.689805] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4904.251931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4904.326858] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4904.328686] LustreError: 184773:0:(llog.c:1657:llog_backup()) MGC192.168.201.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4904.332639] Lustre: 184773:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.124@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4904.380694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.324354] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4908.099103] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4912.611500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.613551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4912.615931] Lustre: Skipped 2 previous similar messages [ 4912.618385] Lustre: Skipped 1 previous similar message [ 4912.620483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4917.729696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4917.732446] Lustre: Skipped 2 previous similar messages [ 4922.848500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4922.851133] Lustre: Skipped 2 previous similar messages [ 4923.359097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.400698] LustreError: 185424:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4923.433738] Lustre: server umount lustre-MDT0000 complete [ 4924.798304] LustreError: 183114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158302 with bad export cookie 3292544114742409414 [ 4924.801432] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4924.803128] LustreError: 183114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4924.875876] LustreError: 185629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4924.878176] LustreError: 185629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4924.928727] Lustre: server umount lustre-MDT0001 complete [ 4936.369111] LustreError: 185829:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4936.371409] LustreError: 185829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4936.420943] Lustre: server umount lustre-OST0000 complete [ 4939.621700] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4943.271965] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4947.252433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4949.859318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.075386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4954.386468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4956.823851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4956.850320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.948623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4956.963896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4957.003268] Lustre: lustre-MDT0000: new disk, initializing [ 4957.032544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.039724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4958.474521] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4962.918027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4962.954168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4962.995090] Lustre: 188164:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4963.019245] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4963.020947] Lustre: Skipped 1 previous similar message [ 4963.022904] LustreError: 188929:0:(llog.c:1657:llog_backup()) MGC192.168.201.124@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4963.027087] LustreError: 188929:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 4963.030145] Lustre: 188929:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.124@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4963.040609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4963.043230] Lustre: Skipped 1 previous similar message [ 4963.083248] Lustre: lustre-MDT0001: new disk, initializing [ 4963.130868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4963.147802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4963.152687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4964.867095] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4968.147883] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4969.515132] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4971.944838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4971.969905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4972.073261] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4972.074297] Lustre: Skipped 1 previous similar message [ 4972.076216] LustreError: 189886:0:(llog.c:1657:llog_backup()) MGC192.168.201.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4972.080180] LustreError: 189886:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 4972.082768] Lustre: 189886:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.124@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4972.086067] Lustre: 189886:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4972.096287] Lustre: lustre-OST0000: new disk, initializing [ 4972.098597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4972.128298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4974.186347] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4977.369385] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4977.673712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4977.677439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4977.723353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4978.920292] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4979.003690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4980.420905] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4980.514242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4982.756377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4982.757200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.759055] LustreError: Skipped 1 previous similar message [ 4982.765154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4982.769308] Lustre: Skipped 1 previous similar message [ 4987.244935] LustreError: 191101:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4987.247134] LustreError: 191101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4987.280934] Lustre: server umount lustre-OST0000 complete [ 4988.895996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4988.898686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.904517] Lustre: Skipped 1 previous similar message [ 4988.907419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4992.992890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.687736] LustreError: 191304:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4994.690082] LustreError: 191304:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4994.716613] Lustre: server umount lustre-MDT0000 complete [ 4996.044785] LustreError: 188156:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158373 with bad export cookie 3292544114742410114 [ 4996.049279] LustreError: 188156:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4996.049930] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4996.166577] Lustre: server umount lustre-MDT0001 complete [ 4999.241681] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:12:55 (1763158375) [ 5004.731642] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5008.310424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.440161] LustreError: 192869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.446693] LustreError: 192869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5008.458299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.866650] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5012.839294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5014.280886] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5015.194848] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5017.287659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5019.361777] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5022.234038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5022.259961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5022.317110] Lustre: lustre-OST0001: new disk, initializing [ 5022.319685] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5024.156152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5024.160248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5024.223227] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5024.548778] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5030.052081] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5033.739354] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5039.380876] Lustre: DEBUG MARKER: ADD OST3 [ 5041.064444] loop: module loaded [ 5041.083980] loop0: detected capacity change from 0 to 204800000 [ 5041.181170] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5041.184549] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5041.188588] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5041.238490] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5043.229105] loop0: detected capacity change from 0 to 204800000 [ 5045.214149] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.243051] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5045.314465] Lustre: lustre-OST0003: new disk, initializing [ 5045.317596] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5045.362994] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5045.366315] Lustre: Skipped 3 previous similar messages [ 5046.215149] Lustre: DEBUG MARKER: STOP OST3 [ 5046.898510] Lustre: Failing over lustre-OST0003 [ 5046.900490] Lustre: lustre-OST0003: Not available for connect from 192.168.201.24@tcp (stopping) [ 5046.902860] Lustre: Skipped 2 previous similar messages [ 5046.922198] LustreError: 197199:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5046.924983] LustreError: 197199:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5046.948043] Lustre: server umount lustre-OST0003 complete [ 5047.316604] LustreError: 194320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.586415] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5051.313077] Lustre: DEBUG MARKER: START OST3 again [ 5051.936615] LustreError: 195403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.940918] LustreError: 195403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5053.456821] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5055.283450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5055.288110] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5055.297357] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5055.520075] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5069.632168] Lustre: Failing over lustre-OST0003 [ 5069.664779] LustreError: 199152:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5069.669715] LustreError: 199152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5069.704257] Lustre: server umount lustre-OST0003 complete [ 5070.816663] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5070.818319] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.820305] LustreError: Skipped 1 previous similar message [ 5070.826462] LustreError: 194320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.835895] LustreError: 194320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5079.010627] LustreError: 194321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5079.022129] LustreError: 194321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5084.131048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.135225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5084.136410] Lustre: Skipped 4 previous similar messages [ 5084.139298] Lustre: Skipped 2 previous similar messages [ 5088.739672] Lustre: server umount lustre-MDT0000 complete [ 5089.249766] LustreError: 195133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.255902] LustreError: 195133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5090.135443] LustreError: 192850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158467 with bad export cookie 3292544114742410982 [ 5090.137361] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5090.140178] LustreError: 192850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5096.432780] Lustre: server umount lustre-MDT0001 complete [ 5097.736818] Lustre: server umount lustre-OST0000 complete [ 5098.986260] Lustre: server umount lustre-OST0001 complete [ 5101.071777] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5103.508335] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5106.939334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5109.428908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5111.455646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5113.742426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5115.981955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5116.003382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.088211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5116.099196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5116.130000] Lustre: lustre-MDT0000: new disk, initializing [ 5116.153664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.156433] Lustre: Skipped 1 previous similar message [ 5116.162676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5117.423511] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5121.456540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5121.484906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5121.526175] Lustre: 202208:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5121.652853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5123.035410] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5126.198387] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.416186] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5129.790288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5129.811045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5131.007270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5131.058573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5131.894223] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5135.045342] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5136.402489] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5136.474570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5137.805787] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5137.882786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5141.477272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5141.477407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5141.482617] Lustre: Skipped 2 previous similar messages [ 5141.483676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5141.485784] LustreError: Skipped 1 previous similar message [ 5141.486993] Lustre: Skipped 5 previous similar messages [ 5144.557101] LustreError: 205146:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5144.559630] LustreError: 205146:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 5144.597401] Lustre: server umount lustre-OST0000 complete [ 5151.962858] Lustre: server umount lustre-MDT0000 complete [ 5153.211581] LustreError: 202200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158530 with bad export cookie 3292544114742413215 [ 5153.216218] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.216450] LustreError: 202200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.333638] Lustre: server umount lustre-MDT0001 complete [ 5154.564962] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:15:31 (1763158531) [ 5159.911589] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5163.197957] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5166.503045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5168.614747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5170.675994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5172.752568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5174.939959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5174.960117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.060195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5175.072219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5175.074439] Lustre: Skipped 3 previous similar messages [ 5175.135169] Lustre: lustre-MDT0000: new disk, initializing [ 5175.136569] Lustre: Skipped 2 previous similar messages [ 5175.169467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5175.172509] Lustre: Skipped 2 previous similar messages [ 5176.542184] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5180.826845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5180.859940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5180.902151] Lustre: 208386:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5181.011989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5182.366801] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5185.654126] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5186.951106] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5189.355852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5189.383910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5190.656709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5191.651886] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5194.941303] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5196.459570] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5196.539578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5198.068985] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5198.162681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5200.865822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5200.866453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.868830] LustreError: Skipped 1 previous similar message [ 5200.873886] Lustre: Skipped 4 previous similar messages [ 5205.985734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5205.987797] Lustre: Skipped 8 previous similar messages [ 5213.151106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5213.169090] LustreError: 211323:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5213.170906] LustreError: 211323:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5213.214687] Lustre: server umount lustre-OST0000 complete [ 5220.636250] Lustre: server umount lustre-MDT0000 complete [ 5221.345478] LustreError: 209171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.349397] LustreError: 209171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5221.963337] LustreError: 208377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158599 with bad export cookie 3292544114742414097 [ 5221.965549] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.969295] LustreError: 208377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5222.103680] Lustre: server umount lustre-MDT0001 complete [ 5225.160109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5226.607602] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5229.483828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5230.785767] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5232.547775] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5233.774182] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5236.046791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5237.975891] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5239.762934] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5261.794310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5267.795111] Lustre: server umount lustre-OST0000 complete [ 5270.496136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.501958] Lustre: Skipped 4 previous similar messages [ 5275.353785] Lustre: server umount lustre-MDT0000 complete [ 5276.705518] LustreError: 212228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158654 with bad export cookie 3292544114742414979 [ 5276.710659] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.711274] LustreError: 212228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5276.838440] Lustre: server umount lustre-MDT0001 complete [ 5372.781836] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5373.748646] Key type lgssc unregistered [ 5373.876333] LNet: 215534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5373.879227] LNetError: 215534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5373.887288] LNet: Removed LNI 192.168.201.124@tcp [ 5374.184108] Key type .llcrypt unregistered [ 5374.185442] Key type ._llcrypt unregistered [ 5382.085821] Key type ._llcrypt registered [ 5382.087485] Key type .llcrypt registered [ 5382.122107] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5388.168102] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5388.540125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5388.591067] alg: No test for adler32 (adler32-zlib) [ 5389.449689] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5389.531540] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5391.111169] Key type lgssc registered [ 5391.530255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5394.557465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5396.702857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5398.655533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5400.703634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5403.129408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5403.152767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.159713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.266916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5404.283477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5404.322626] Lustre: lustre-MDT0000: new disk, initializing [ 5404.357971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.365702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5405.738148] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5410.069207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.089124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5410.122989] Lustre: 218835:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5410.153867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5410.155973] Lustre: Skipped 1 previous similar message [ 5410.190210] Lustre: lustre-MDT0001: new disk, initializing [ 5410.229405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5410.243745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5410.248796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5411.627685] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5414.717034] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5415.857529] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5418.153099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5418.183276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5418.303531] Lustre: lustre-OST0000: new disk, initializing [ 5418.305358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5418.344108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.209347] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5423.259688] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.612293] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5425.143567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5425.146561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5425.178463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5425.707245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5427.068591] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5427.155619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5430.244565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.244834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5430.249054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5430.249787] LustreError: Skipped 1 previous similar message [ 5433.901618] LustreError: 221779:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5433.941476] Lustre: server umount lustre-OST0000 complete [ 5435.360230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.360772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5435.366078] Lustre: Skipped 2 previous similar messages [ 5435.370219] Lustre: Skipped 2 previous similar messages [ 5440.480467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5440.483406] Lustre: Skipped 1 previous similar message [ 5441.282305] LustreError: 221980:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5441.285311] LustreError: 221980:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5441.321193] Lustre: server umount lustre-MDT0000 complete [ 5442.557452] LustreError: 218825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158819 with bad export cookie 11461397750666714842 [ 5442.561378] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5442.561687] LustreError: 218825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5442.627514] LustreError: 222182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5442.629647] LustreError: 222182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5442.690669] Lustre: server umount lustre-MDT0001 complete [ 5448.313616] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5451.617769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.771136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.006686] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5455.770875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5455.885238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5457.087425] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5457.984666] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5460.172603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5462.177735] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5465.076379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5465.101990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5465.161490] Lustre: lustre-OST0001: new disk, initializing [ 5465.164361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5465.201595] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5465.203688] Lustre: Skipped 1 previous similar message [ 5467.150390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5467.154233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5467.198034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5467.319295] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5472.659705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5475.171081] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5484.198182] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:21:00 (1763158860) [ 5486.560089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5486.563743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.569717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.007988] LustreError: 226737:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5491.010142] LustreError: 226737:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5491.038991] Lustre: server umount lustre-MDT0000 complete [ 5492.304639] LustreError: 225620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158869 with bad export cookie 11461397750666715710 [ 5492.308554] LustreError: 225620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5492.309178] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5492.432689] Lustre: server umount lustre-MDT0001 complete [ 5504.115035] LustreError: 227138:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5504.116890] LustreError: 227138:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5504.134539] Lustre: server umount lustre-OST0000 complete [ 5515.569317] LustreError: 227340:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5515.571975] LustreError: 227340:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5515.617102] Lustre: server umount lustre-OST0001 complete [ 5517.646074] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5520.596731] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5523.687723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5525.637355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5527.530264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5529.523147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5531.692710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5531.713155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.795258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5531.809563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5531.846841] Lustre: lustre-MDT0000: new disk, initializing [ 5531.872251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5531.879550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5533.212688] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5537.379269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5537.406157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5537.441631] Lustre: 229521:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5537.550264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5538.783333] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5541.734273] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5542.872567] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5545.083813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5545.109241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5545.202446] Lustre: lustre-OST0000: new disk, initializing [ 5545.203828] Lustre: Skipped 1 previous similar message [ 5545.206532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5545.209025] Lustre: Skipped 2 previous similar messages [ 5546.360109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5546.363504] Lustre: Skipped 1 previous similar message [ 5546.365724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5546.404767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5547.079621] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5550.047591] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5551.291611] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5551.361579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5552.630252] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5552.719601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5555.437818] LustreError: 232458:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5555.440135] LustreError: 232458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5555.477399] Lustre: server umount lustre-OST0000 complete [ 5558.239809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5558.242702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.247194] Lustre: Skipped 3 previous similar messages [ 5558.249376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5558.251584] Lustre: Skipped 4 previous similar messages [ 5566.944290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5566.946328] Lustre: Skipped 3 previous similar messages [ 5571.039144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5571.101116] Lustre: server umount lustre-MDT0000 complete [ 5572.063924] LustreError: 229533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.068583] LustreError: 229533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5572.263150] LustreError: 229513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158949 with bad export cookie 11461397750666717810 [ 5572.267161] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.267536] LustreError: 229513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5581.841806] Lustre: server umount lustre-MDT0001 complete [ 5584.693041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.851685] LustreError: 233339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.881771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.883600] Lustre: Skipped 2 previous similar messages [ 5586.168500] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5589.043058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5590.490770] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5592.200686] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5593.412318] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5595.590058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5595.618503] LustreError: 234937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5597.549514] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5599.270403] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5743.074376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.078299] Lustre: Skipped 1 previous similar message [ 5743.079545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.079839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5743.083925] Lustre: Skipped 1 previous similar message [ 5748.524074] LustreError: 235626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5748.526774] LustreError: 235626:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5748.566899] Lustre: server umount lustre-OST0000 complete [ 5753.312576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5753.312696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.319255] Lustre: Skipped 1 previous similar message [ 5755.998856] Lustre: server umount lustre-MDT0000 complete [ 5757.310239] LustreError: 234058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159134 with bad export cookie 11461397750666718685 [ 5757.312623] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5757.314427] LustreError: 234058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5757.443135] Lustre: server umount lustre-MDT0001 complete [ 5863.813309] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5864.880692] Key type lgssc unregistered [ 5865.026401] LNet: 236595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5865.028967] LNetError: 236595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5865.036430] LNet: Removed LNI 192.168.201.124@tcp [ 5865.392118] Key type .llcrypt unregistered [ 5865.393196] Key type ._llcrypt unregistered [ 5874.187731] Key type ._llcrypt registered [ 5874.188733] Key type .llcrypt registered [ 5874.234713] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 5880.601888] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5881.181351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5881.199989] alg: No test for adler32 (adler32-zlib) [ 5882.059770] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5882.146638] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5883.727192] Key type lgssc registered [ 5884.131407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5887.160914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5889.436565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5891.609704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5893.815944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5899.391303] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5903.110204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5903.131678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5903.135738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.226429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5904.239969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5904.276105] Lustre: lustre-MDT0000: new disk, initializing [ 5904.300965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.307357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5905.609591] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5909.869494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5909.896449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5909.938904] Lustre: 240591:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5909.971034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5909.974081] Lustre: Skipped 1 previous similar message [ 5910.020584] Lustre: lustre-MDT0001: new disk, initializing [ 5910.064445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5910.081190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5910.085923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5911.429963] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5913.764180] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5916.157446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5916.186702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5916.293190] Lustre: lustre-OST0000: new disk, initializing [ 5916.295379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5916.329777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.435879] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5921.797123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5921.800259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5921.865831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5922.852449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5922.874322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5922.928310] Lustre: lustre-OST0001: new disk, initializing [ 5922.930949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5922.966418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5924.921397] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5929.986293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5929.990838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5930.019859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5930.350783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5935.053459] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5945.315031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.316617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.321240] Lustre: Skipped 1 previous similar message [ 5945.326097] Lustre: Skipped 3 previous similar messages [ 5949.199724] LustreError: 244116:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5949.238132] Lustre: server umount lustre-MDT0000 complete [ 5950.536367] LustreError: 241372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159327 with bad export cookie 931341028955081467 [ 5950.539090] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5950.540064] LustreError: 241372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5950.603413] LustreError: 244317:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5950.605319] LustreError: 244317:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5950.662886] Lustre: server umount lustre-MDT0001 complete [ 5961.720906] LustreError: 244517:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5961.724015] LustreError: 244517:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5961.746669] Lustre: server umount lustre-OST0000 complete [ 5973.299636] LustreError: 244719:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5973.302136] LustreError: 244719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5973.350310] Lustre: server umount lustre-OST0001 complete [ 5977.944726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5980.763679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5981.110918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5984.065164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5986.959822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5987.320849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5993.786945] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5997.638299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.644458] Lustre: lustre-MDT0000: reset Object Index mappings [ 5997.814586] LustreError: 248026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.821769] LustreError: 248026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5997.834203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.251408] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6002.166276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6002.173803] Lustre: lustre-MDT0001: reset Object Index mappings [ 6003.761650] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6004.787351] Lustre: 249135:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6007.062282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6007.208649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.211591] Lustre: Skipped 1 previous similar message [ 6008.232782] LustreError: 249490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.238075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6009.280527] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6012.195500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6013.303075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6014.190538] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6017.306780] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6018.464588] Lustre: 250981:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6028.255968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6028.258185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.261254] Lustre: Skipped 2 previous similar messages [ 6028.262939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6030.723021] LustreError: 251086:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6030.725208] LustreError: 251086:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6030.755753] Lustre: server umount lustre-MDT0000 complete [ 6032.045510] LustreError: 248754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159409 with bad export cookie 931341028955088243 [ 6032.049252] LustreError: 248754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6032.049956] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6032.169582] Lustre: server umount lustre-MDT0001 complete [ 6043.059160] LustreError: 251488:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6043.061142] LustreError: 251488:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6043.078506] Lustre: server umount lustre-OST0000 complete [ 6054.683114] Lustre: server umount lustre-OST0001 complete [ 6061.220771] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:30:37 (1763159437) [ 6066.713377] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6069.915448] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6073.540164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6075.785408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6077.876565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6080.074282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6082.427015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6082.447409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.536591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6082.550911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6082.594157] Lustre: lustre-MDT0000: new disk, initializing [ 6082.618622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.620497] Lustre: Skipped 1 previous similar message [ 6082.625799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6083.923159] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6088.237626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6088.266962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6088.302173] Lustre: 255209:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6088.408679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6089.815023] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6093.091582] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6094.404252] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6096.841454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6096.875906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6097.005270] Lustre: lustre-OST0000: new disk, initializing [ 6097.007942] Lustre: Skipped 1 previous similar message [ 6097.011418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6097.014233] Lustre: Skipped 2 previous similar messages [ 6099.009601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6099.012067] Lustre: Skipped 1 previous similar message [ 6099.014038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6099.065127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6099.156160] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6102.379917] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6103.869242] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6103.952156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6105.322197] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6105.400252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6108.080259] LustreError: 258148:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6108.083341] LustreError: 258148:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6108.127469] Lustre: server umount lustre-OST0000 complete [ 6109.664391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.667814] Lustre: Skipped 3 previous similar messages [ 6109.669569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6109.671536] Lustre: Skipped 3 previous similar messages [ 6114.271896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6114.274033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.277671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6118.911125] Lustre: server umount lustre-MDT0000 complete [ 6119.904649] LustreError: 256007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.909835] LustreError: 256007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6120.260554] LustreError: 255199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159497 with bad export cookie 931341028955090336 [ 6120.265830] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.266028] LustreError: 255199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6130.717303] Lustre: server umount lustre-MDT0001 complete [ 6136.794976] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6140.577953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.738373] LustreError: 259725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.756682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6140.759941] Lustre: Skipped 2 previous similar messages [ 6142.139526] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6145.021208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6146.507825] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6147.493258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6149.731429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6151.828952] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6154.760177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6154.788996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6154.854179] Lustre: lustre-OST0001: new disk, initializing [ 6154.856459] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6156.428792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6156.433609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6156.466762] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6156.955250] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6161.464363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6162.685722] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6169.512861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6169.519829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6172.129948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6172.132132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6172.135718] Lustre: Skipped 3 previous similar messages [ 6172.137310] Lustre: Skipped 3 previous similar messages [ 6177.411780] LustreError: 263229:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6177.415644] LustreError: 263229:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6177.447458] Lustre: server umount lustre-MDT0000 complete [ 6178.712352] LustreError: 259706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159556 with bad export cookie 931341028955091218 [ 6178.714991] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.718061] LustreError: 259706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6178.840657] Lustre: server umount lustre-MDT0001 complete [ 6190.593830] Lustre: server umount lustre-OST0000 complete [ 6202.146766] Lustre: server umount lustre-OST0001 complete [ 6207.903873] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6211.382689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.524546] LustreError: 265007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.542883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.544767] Lustre: Skipped 3 previous similar messages [ 6212.906013] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6215.891974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6217.437298] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6218.504219] Lustre: 266116:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6220.788092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6221.986763] LustreError: 266472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.995994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6222.967140] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6225.784811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6226.926651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6227.859305] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6231.053563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6232.195917] Lustre: 267967:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6238.963279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6238.966406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6241.960380] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:33:38 (1763159618) [ 6257.120495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6257.123854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.130107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6257.133955] Lustre: Skipped 5 previous similar messages [ 6259.520841] LustreError: 268677:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6259.522710] LustreError: 268677:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6259.552047] Lustre: server umount lustre-MDT0000 complete [ 6260.759359] LustreError: 264986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159638 with bad export cookie 931341028955097504 [ 6260.762571] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6260.762739] LustreError: 264986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6262.752107] LustreError: 265722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.759071] LustreError: 265722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6275.039103] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6275.121265] Lustre: server umount lustre-MDT0001 complete [ 6282.430632] Lustre: server umount lustre-OST0000 complete [ 6289.751431] Lustre: server umount lustre-OST0001 complete [ 6291.718432] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6294.866373] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6298.751235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6300.937541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6302.996326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6305.237253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6310.947324] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6314.627670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6314.654740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.749672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6314.764235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6314.805396] Lustre: lustre-MDT0000: new disk, initializing [ 6314.837266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6316.208470] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6320.328321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6320.353685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6320.390542] Lustre: 272172:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6320.498597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6321.807033] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6326.400390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6326.422349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6326.526140] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6326.528993] Lustre: Skipped 9 previous similar messages [ 6327.683173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6328.543195] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6332.645173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6332.673827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6333.934458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6334.696515] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6339.102412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6349.318561] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 17:35:25 (1763159725) [ 6351.327893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6351.330422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.334082] Lustre: Skipped 5 previous similar messages [ 6356.137675] Lustre: server umount lustre-MDT0000 complete [ 6357.382563] LustreError: 272164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159734 with bad export cookie 931341028955105372 [ 6357.385686] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6357.389025] LustreError: 272164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6357.527798] Lustre: server umount lustre-MDT0001 complete [ 6368.708252] Lustre: server umount lustre-OST0000 complete [ 6380.195340] Lustre: server umount lustre-OST0001 complete [ 6385.881905] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6388.869607] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6392.021257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6394.123881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.175198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6398.407549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.046275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.073044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6401.176129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6401.179812] Lustre: Skipped 2 previous similar messages [ 6401.190732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6401.192842] Lustre: Skipped 4 previous similar messages [ 6401.237705] Lustre: lustre-MDT0000: new disk, initializing [ 6401.239911] Lustre: Skipped 3 previous similar messages [ 6401.271396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.274921] Lustre: Skipped 7 previous similar messages [ 6401.282561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6401.286800] Lustre: Skipped 3 previous similar messages [ 6402.662972] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6406.935529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6406.959120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6406.994386] Lustre: 279235:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6407.099351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6407.101588] Lustre: Skipped 2 previous similar messages [ 6408.481294] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6411.490500] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6412.626626] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6414.924408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6414.947161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6416.548795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6416.908606] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6420.017814] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6420.777399] Lustre: Mounted lustre-client [ 6421.087203] LustreError: 281735:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e774495e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6421.092246] LustreError: 281735:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6421.094016] LustreError: 281735:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 6421.112182] Lustre: Unmounted lustre-client [ 6426.082391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.083058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.089298] Lustre: Skipped 4 previous similar messages [ 6426.091124] Lustre: Skipped 5 previous similar messages [ 6430.224892] Lustre: server umount lustre-OST0000 complete [ 6432.735983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6437.655554] Lustre: server umount lustre-MDT0000 complete [ 6438.942485] LustreError: 279228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159816 with bad export cookie 931341028955106898 [ 6438.945851] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6438.946467] LustreError: 279228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6439.066350] Lustre: server umount lustre-MDT0001 complete [ 6442.664044] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6443.711081] Key type lgssc unregistered [ 6474.591419] LNet: 283014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6474.593728] LNetError: 283014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6551.397597] LNet: Removed LNI 192.168.201.124@tcp [ 6551.742112] Key type .llcrypt unregistered [ 6551.743193] Key type ._llcrypt unregistered [ 6558.086235] Key type ._llcrypt registered [ 6558.087866] Key type .llcrypt registered [ 6558.141541] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6564.987870] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6565.425972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6565.497692] alg: No test for adler32 (adler32-zlib) [ 6566.372486] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6566.468963] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6568.047175] Key type lgssc registered [ 6568.399319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6571.884775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6574.049774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6576.233296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6578.407058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.013784] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:39:15 (1763159955) [ 6581.194563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6581.214177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6581.221298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.309393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6582.321358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6582.358133] Lustre: lustre-MDT0000: new disk, initializing [ 6582.386793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.394685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6583.918950] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6588.866243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.899606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6588.957880] Lustre: 286183:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6588.994966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6588.998025] Lustre: Skipped 1 previous similar message [ 6589.037748] Lustre: lustre-MDT0001: new disk, initializing [ 6589.081141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6589.095911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6589.099990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6590.526567] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6593.603129] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6594.776334] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6597.060349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.083438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6597.179257] Lustre: lustre-OST0000: new disk, initializing [ 6597.181586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6597.210760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.054803] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6602.113795] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6603.775640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6603.778953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6603.818501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6613.986506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6613.988899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6613.990312] LustreError: Skipped 1 previous similar message [ 6613.995880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6613.998322] Lustre: Skipped 1 previous similar message [ 6617.200130] LustreError: 288625:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6617.256167] Lustre: server umount lustre-OST0000 complete [ 6619.106240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.107705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6619.112847] Lustre: Skipped 2 previous similar messages [ 6619.118712] Lustre: Skipped 1 previous similar message [ 6624.225219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.229247] Lustre: Skipped 1 previous similar message [ 6625.022461] LustreError: 288826:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6625.024294] LustreError: 288826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6625.049918] Lustre: server umount lustre-MDT0000 complete [ 6626.444266] LustreError: 286175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160003 with bad export cookie 16855482865565053366 [ 6626.447948] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6626.449631] LustreError: 286175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6626.552124] LustreError: 289029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6626.555336] LustreError: 289029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6626.616402] Lustre: server umount lustre-MDT0001 complete [ 6630.286681] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6631.583054] Key type lgssc unregistered [ 6631.702398] LNet: 289561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6631.705275] LNetError: 289561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6631.717410] LNet: Removed LNI 192.168.201.124@tcp [ 6632.032461] Key type .llcrypt unregistered [ 6632.033683] Key type ._llcrypt unregistered [ 6667.862463] Key type ._llcrypt registered [ 6667.865653] Key type .llcrypt registered [ 6667.904648] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6668.001432] Key type .llcrypt unregistered [ 6668.002538] Key type ._llcrypt unregistered [ 6695.468945] Key type ._llcrypt registered [ 6695.470925] Key type .llcrypt registered [ 6695.508885] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6695.615446] Key type .llcrypt unregistered [ 6695.617105] Key type ._llcrypt unregistered [ 6706.454648] Key type ._llcrypt registered [ 6706.456360] Key type .llcrypt registered [ 6706.499517] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6706.610183] Key type .llcrypt unregistered [ 6706.611200] Key type ._llcrypt unregistered [ 6718.005130] Key type ._llcrypt registered [ 6718.006173] Key type .llcrypt registered [ 6718.039664] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6718.124263] Key type .llcrypt unregistered [ 6718.125301] Key type ._llcrypt unregistered [ 6744.392853] Key type ._llcrypt registered [ 6744.393935] Key type .llcrypt registered [ 6744.427816] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6744.514375] Key type .llcrypt unregistered [ 6744.515419] Key type ._llcrypt unregistered [ 6755.289916] Key type ._llcrypt registered [ 6755.291021] Key type .llcrypt registered [ 6755.323693] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6755.413350] Key type .llcrypt unregistered [ 6755.414385] Key type ._llcrypt unregistered [ 6766.480817] Key type ._llcrypt registered [ 6766.481910] Key type .llcrypt registered [ 6766.516832] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6766.619482] Key type .llcrypt unregistered [ 6766.621385] Key type ._llcrypt unregistered [ 6777.777637] Key type ._llcrypt registered [ 6777.779069] Key type .llcrypt registered [ 6777.822795] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6777.919218] Key type .llcrypt unregistered [ 6777.920113] Key type ._llcrypt unregistered [ 6788.854764] Key type ._llcrypt registered [ 6788.855992] Key type .llcrypt registered [ 6788.895920] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6789.002360] Key type .llcrypt unregistered [ 6789.003561] Key type ._llcrypt unregistered [ 6801.535992] Key type ._llcrypt registered [ 6801.536972] Key type .llcrypt registered [ 6801.584610] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6801.699231] Key type .llcrypt unregistered [ 6801.700853] Key type ._llcrypt unregistered [ 6824.539755] Key type ._llcrypt registered [ 6824.540798] Key type .llcrypt registered [ 6824.582869] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6824.697425] Key type .llcrypt unregistered [ 6824.698463] Key type ._llcrypt unregistered [ 6835.433897] Key type ._llcrypt registered [ 6835.435099] Key type .llcrypt registered [ 6835.473854] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6835.575206] Key type .llcrypt unregistered [ 6835.576606] Key type ._llcrypt unregistered [ 6846.338174] Key type ._llcrypt registered [ 6846.339308] Key type .llcrypt registered [ 6846.382587] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6846.492268] Key type .llcrypt unregistered [ 6846.493546] Key type ._llcrypt unregistered [ 6865.006552] Key type ._llcrypt registered [ 6865.007841] Key type .llcrypt registered [ 6865.045349] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 6871.028843] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6871.402866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6871.452689] alg: No test for adler32 (adler32-zlib) [ 6872.363060] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6872.449587] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6874.031156] Key type lgssc registered [ 6874.425639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6877.457461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6879.690115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6881.899246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.150255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.733879] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6695 sec ========== 17:44:21 (1763160261) [ 6885.325943] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:44:21 (1763160261) === [ 6886.437723] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:44:23 (1763160263) === [ 6899.498529] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6900.559234] Key type lgssc unregistered [ 6900.692510] LNet: 311254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6900.694955] LNetError: 311254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6900.706463] LNet: Removed LNI 192.168.201.124@tcp [ 6901.017432] Key type .llcrypt unregistered [ 6901.018391] Key type ._llcrypt unregistered