[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 410441637 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002157] x2apic enabled [ 0.002942] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005855] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006008] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008070] LSM: Security Framework initializing [ 0.009025] Yama: becoming mindful. [ 0.010019] SELinux: Initializing. [ 0.011037] *** VALIDATE selinux *** [ 0.017641] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021335] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022080] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023060] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024054] *** VALIDATE tmpfs *** [ 0.025227] *** VALIDATE proc *** [ 0.026030] *** VALIDATE cgroup *** [ 0.026680] *** VALIDATE cgroup2 *** [ 0.027165] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029014] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031017] Spectre V2 : User space: Vulnerable [ 0.032003] Speculative Store Bypass: Vulnerable [ 0.034366] debug: unmapping init [mem 0xffffffffb0e59000-0xffffffffb0e60fff] [ 0.036748] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037435] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038009] ... version: 2 [ 0.038783] ... bit width: 48 [ 0.039006] ... generic registers: 4 [ 0.039823] ... value mask: 0000ffffffffffff [ 0.040006] ... max period: 00007fffffffffff [ 0.041006] ... fixed-purpose events: 3 [ 0.041778] ... event mask: 000000070000000f [ 0.042136] rcu: Hierarchical SRCU implementation. [ 0.044130] smp: Bringing up secondary CPUs ... [ 0.045375] x86: Booting SMP configuration: [ 0.046014] .... node #0, CPUs: #1 #2 #3 [ 0.050530] smp: Brought up 1 node, 4 CPUs [ 0.051969] smpboot: Max logical packages: 1 [ 0.052020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241027] node 0 deferred pages initialised in 188ms [ 0.243226] devtmpfs: initialized [ 0.244163] x86/mm: Memory block size: 128MB [ 0.246194] gcov: version magic: 0x41383552 [ 0.248107] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.251294] pinctrl core: initialized pinctrl subsystem [ 0.253082] [ 0.253389] ************************************************************* [ 0.254005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256006] ** ** [ 0.257005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.259006] ** ** [ 0.260005] ** This means that this kernel is built to expose internal ** [ 0.261005] ** IOMMU data structures, which may compromise security on ** [ 0.263006] ** your system. ** [ 0.264005] ** ** [ 0.266007] ** If you see this message and you are not debugging the ** [ 0.267005] ** kernel, report this immediately to your vendor! ** [ 0.269006] ** ** [ 0.270005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271005] ************************************************************* [ 0.273550] NET: Registered protocol family 16 [ 0.274306] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279310] cpuidle: using governor menu [ 0.280389] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281339] PCI: Using configuration type 1 for base access [ 0.283092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.290102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295097] cryptd: max_cpu_qlen set to 1000 [ 0.297174] ACPI: Added _OSI(Module Device) [ 0.298082] ACPI: Added _OSI(Processor Device) [ 0.300012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.302011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.307844] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.316675] ACPI: Interpreter enabled [ 0.318051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320011] ACPI: Using IOAPIC for interrupt routing [ 0.322078] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.325393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.337352] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.339024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.342012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.346053] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.351577] acpiphp: Slot [2] registered [ 0.352084] acpiphp: Slot [5] registered [ 0.352960] acpiphp: Slot [6] registered [ 0.354069] acpiphp: Slot [7] registered [ 0.354942] acpiphp: Slot [8] registered [ 0.356062] acpiphp: Slot [9] registered [ 0.356936] acpiphp: Slot [10] registered [ 0.358079] acpiphp: Slot [3] registered [ 0.359073] acpiphp: Slot [4] registered [ 0.361070] acpiphp: Slot [11] registered [ 0.362100] acpiphp: Slot [12] registered [ 0.364065] acpiphp: Slot [13] registered [ 0.365076] acpiphp: Slot [14] registered [ 0.367072] acpiphp: Slot [15] registered [ 0.369084] acpiphp: Slot [16] registered [ 0.370061] acpiphp: Slot [17] registered [ 0.372069] acpiphp: Slot [18] registered [ 0.373067] acpiphp: Slot [19] registered [ 0.375072] acpiphp: Slot [20] registered [ 0.376069] acpiphp: Slot [21] registered [ 0.378085] acpiphp: Slot [22] registered [ 0.379087] acpiphp: Slot [23] registered [ 0.381067] acpiphp: Slot [24] registered [ 0.383059] acpiphp: Slot [25] registered [ 0.383958] acpiphp: Slot [26] registered [ 0.384045] acpiphp: Slot [27] registered [ 0.384956] acpiphp: Slot [28] registered [ 0.386055] acpiphp: Slot [29] registered [ 0.386949] acpiphp: Slot [30] registered [ 0.388049] acpiphp: Slot [31] registered [ 0.388939] PCI host bridge to bus 0000:00 [ 0.390011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.392007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.394008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.395008] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.397015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.398159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399761] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.401800] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.412036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.414009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.415007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.416007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.418496] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.420574] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.423251] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.427008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.437013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.446190] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.453012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463010] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470156] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487989] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494170] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498007] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502008] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512008] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518007] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.521007] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.525008] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535009] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.541555] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.545007] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548007] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.556024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.565202] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.569007] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.572010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.580007] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.586580] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.588244] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.591205] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.592185] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.594105] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.597104] iommu: Default domain type: Passthrough [ 0.598284] SCSI subsystem initialized [ 0.599079] ACPI: bus type USB registered [ 0.599994] usbcore: registered new interface driver usbfs [ 0.601029] usbcore: registered new interface driver hub [ 0.602031] usbcore: registered new device driver usb [ 0.603088] pps_core: LinuxPPS API ver. 1 registered [ 0.604006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.606017] PTP clock support registered [ 0.607124] EDAC MC: Ver: 3.0.0 [ 0.608495] PCI: Using ACPI for IRQ routing [ 0.609881] NetLabel: Initializing [ 0.610010] NetLabel: domain hash size = 128 [ 0.611008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.612092] NetLabel: unlabeled traffic allowed by default [ 0.613101] vgaarb: loaded [ 0.614134] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.615006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.621267] clocksource: Switched to clocksource kvm-clock [ 0.699850] VFS: Disk quotas dquot_6.6.0 [ 0.700921] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.703361] *** VALIDATE ramfs *** [ 0.704489] *** VALIDATE hugetlbfs *** [ 0.706594] pnp: PnP ACPI init [ 0.709280] pnp: PnP ACPI: found 6 devices [ 0.723529] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.726236] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.727670] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.729146] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.730717] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.732225] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.734766] NET: Registered protocol family 2 [ 0.737115] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.740632] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.742927] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.746802] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.748789] TCP: Hash tables configured (established 65536 bind 65536) [ 0.750544] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.752381] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754035] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.755676] NET: Registered protocol family 1 [ 0.757304] RPC: Registered named UNIX socket transport module. [ 0.758550] RPC: Registered udp transport module. [ 0.759483] RPC: Registered tcp transport module. [ 0.760483] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.761815] NET: Registered protocol family 44 [ 0.762777] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.764076] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.765339] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.766712] PCI: CLS 0 bytes, default 64 [ 0.767768] Unpacking initramfs... [ 2.027851] debug: unmapping init [mem 0xffff96ddbcc54000-0xffff96ddbffbffff] [ 2.030399] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.031788] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.033549] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.514881] Initialise system trusted keyrings [ 2.515961] Key type blacklist registered [ 2.517183] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.523242] zbud: loaded [ 2.525190] *** VALIDATE nfs *** [ 2.525854] *** VALIDATE nfs4 *** [ 2.526825] pstore: using deflate compression [ 2.528519] Platform Keyring initialized [ 2.599447] NET: Registered protocol family 38 [ 2.601262] Key type asymmetric registered [ 2.602779] Asymmetric key parser 'x509' registered [ 2.604690] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.607593] io scheduler mq-deadline registered [ 2.609548] io scheduler kyber registered [ 2.611383] io scheduler bfq registered [ 2.613273] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.616452] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.619686] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.622928] ACPI: Power Button [PWRF] [ 2.628727] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.636131] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.647956] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.655257] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.666942] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.692128] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.720144] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.724964] Non-volatile memory driver v1.3 [ 2.726477] Linux agpgart interface v0.103 [ 2.754628] virtio_blk virtio1: [vda] 134224 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.757608] vda: detected capacity change from 0 to 68722688 [ 2.771294] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.774594] vdb: detected capacity change from 0 to 1073741824 [ 2.789181] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.792310] vdc: detected capacity change from 0 to 2621440000 [ 2.806481] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.809422] vdd: detected capacity change from 0 to 2621440000 [ 2.821537] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.823359] vde: detected capacity change from 0 to 4294967296 [ 2.833224] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.834950] vdf: detected capacity change from 0 to 4294967296 [ 2.838832] libphy: Fixed MDIO Bus: probed [ 2.841941] usbcore: registered new interface driver usbserial_generic [ 2.843402] usbserial: USB Serial support registered for generic [ 2.844824] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.847475] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.848619] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.850132] mousedev: PS/2 mouse device common for all mice [ 2.853222] rtc_cmos 00:05: RTC can wake from S4 [ 2.853340] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.855444] rtc_cmos 00:05: registered as rtc0 [ 2.859301] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.860346] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.861509] intel_pstate: CPU model not supported [ 2.865957] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.870690] hid: raw HID events driver (C) Jiri Kosina [ 2.872046] usbcore: registered new interface driver usbhid [ 2.873319] usbhid: USB HID core driver [ 2.874307] drop_monitor: Initializing network drop monitor service [ 2.875874] Initializing XFRM netlink socket [ 2.877207] NET: Registered protocol family 10 [ 2.879109] Segment Routing with IPv6 [ 2.880135] NET: Registered protocol family 17 [ 2.881538] mpls_gso: MPLS GSO support [ 2.884948] RAS: Correctable Errors collector initialized. [ 2.886332] AVX version of gcm_enc/dec engaged. [ 2.887426] AES CTR mode by8 optimization enabled [ 2.934647] sched_clock: Marking stable (2934626332, 0)->(3541852596, -607226264) [ 2.936947] registered taskstats version 1 [ 2.938204] Loading compiled-in X.509 certificates [ 2.939456] zswap: loaded using pool lzo/zbud [ 2.955058] Key type big_key registered [ 2.962353] Key type encrypted registered [ 2.963541] ima: No TPM chip found, activating TPM-bypass! [ 2.964977] ima: Allocated hash algorithm: sha1 [ 2.966149] ima: No architecture policies found [ 2.967334] evm: Initialising EVM extended attributes: [ 2.968790] evm: security.selinux [ 2.969581] evm: security.ima [ 2.970316] evm: security.capability [ 2.971224] evm: HMAC attrs: 0x1 [ 2.972757] rtc_cmos 00:05: setting system clock to 2026-02-07 06:45:14 UTC (1770446714) [ 2.976990] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 2.978904] debug: unmapping init [mem 0xffffffffb0b82000-0xffffffffb0e58fff] [ 2.986074] Write protecting the kernel read-only data: 28672k [ 2.988192] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 2.989848] debug: unmapping init [mem 0xffffffffafb14000-0xffffffffafbfffff] [ 3.014223] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.019724] systemd[1]: Detected virtualization kvm. [ 3.021968] systemd[1]: Detected architecture x86-64. [ 3.024221] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.050461] systemd[1]: No hostname configured. [ 3.051723] systemd[1]: Set hostname to . [ 3.053251] random: systemd: uninitialized urandom read (16 bytes read) [ 3.054929] systemd[1]: Initializing machine ID from random generator. [ 3.081468] random: ln: uninitialized urandom read (6 bytes read) [ 3.150685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.152697] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.155893] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.159183] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.597302] device-mapper: uevent: version 1.0.3 [ 3.599383] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.261968] random: fast init done [ 4.305317] virtio_net virtio0 ens2: renamed from eth0 [ 4.321730] scsi host0: ata_piix [ 4.346357] scsi host1: ata_piix [ 4.347963] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.350236] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.766509] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.475485] random: crng init done [ 9.476542] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.769415] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.941170] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.203427] SELinux: Disabled at runtime. [ 11.261905] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.269236] systemd[1]: Detected virtualization kvm. [ 11.271172] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.638209] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.641564] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.644184] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.647942] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.654932] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.661684] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.669576] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ 11.713095] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.007584] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.285705] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.292841] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.416847] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.429954] EDAC sbridge: Ver: 1.1.2 [ 13.667988] Key type dns_resolver registered [ 13.966881] NFS: Registering the id_resolver key type [ 13.967981] Key type id_resolver registered [ 13.968991] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg146-server login: [ 35.756902] libcfs: loading out-of-tree module taints kernel. [ 35.772024] Key type ._llcrypt registered [ 35.773030] Key type .llcrypt registered [ 35.807359] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 42.928553] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 43.416928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 43.422771] alg: No test for adler32 (adler32-zlib) [ 44.369251] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 44.641487] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 46.247226] Key type lgssc registered [ 46.829980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 52.834613] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.643598] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 69.694037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 69.702493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 70.792222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 70.802786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 70.839201] Lustre: lustre-MDT0000: new disk, initializing [ 70.862026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 70.868072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 72.026818] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.506942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 77.562338] Lustre: 6485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 77.580926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 77.584105] Lustre: Skipped 1 previous similar message [ 77.632165] Lustre: lustre-MDT0001: new disk, initializing [ 77.654297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 77.663439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 77.668622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.042881] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 81.399409] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 84.736974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 84.869500] Lustre: lustre-OST0000: new disk, initializing [ 84.871713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 84.889551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 86.877769] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 92.898728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 92.943550] Lustre: lustre-OST0001: new disk, initializing [ 92.945406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 92.951205] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 92.954453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 92.957148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 92.973155] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 92.990573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 94.976741] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.276247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 98.278930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 98.294666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 100.856437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 105.754523] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 113.133887] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing check_logdir /tmp/testlogs/ [ 114.517424] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing yml_node [ 116.077874] Lustre: DEBUG MARKER: Client: 2.17.50.44 [ 116.995699] Lustre: DEBUG MARKER: MDS: 2.17.50.44 [ 117.892959] Lustre: DEBUG MARKER: OSS: 2.17.50.44 [ 118.424155] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Feb 7 01:47:08 EST 2026 [ 125.964744] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 126.712845] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 139.232193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 139.232511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 139.234210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 139.240963] Lustre: Skipped 1 previous similar message [ 139.255142] Lustre: Skipped 3 previous similar messages [ 144.352037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 144.354071] Lustre: Skipped 3 previous similar messages [ 144.841511] LustreError: 12177:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 144.881701] Lustre: server umount lustre-MDT0000 complete [ 146.462736] LustreError: 6475:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770446857 with bad export cookie 14055650397845297020 [ 146.465797] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 146.470512] LustreError: 6475:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 149.471981] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.472473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.475603] Lustre: Skipped 3 previous similar messages [ 152.592349] LustreError: 12379:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 152.596699] LustreError: 12379:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 152.675885] Lustre: server umount lustre-MDT0001 complete [ 154.610101] LustreError: 12581:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 154.612920] LustreError: 12581:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 154.634006] Lustre: server umount lustre-OST0000 complete [ 156.358364] Lustre: server umount lustre-OST0001 complete [ 158.795683] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 162.235061] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 175.968829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.112474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.131295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.184341] Lustre: lustre-MDT0000: new disk, initializing [ 176.213053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.223262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.689948] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 182.418788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 182.482846] Lustre: 14948:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.621189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.922589] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 187.494941] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 188.937501] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 191.974196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.120551] Lustre: lustre-OST0000: new disk, initializing [ 192.122156] Lustre: Skipped 1 previous similar message [ 192.123932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.125951] Lustre: Skipped 2 previous similar messages [ 192.158161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.161356] Lustre: Skipped 1 previous similar message [ 193.602362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.607402] Lustre: Skipped 1 previous similar message [ 193.609887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.652885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.426161] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 198.163686] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 199.797191] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 199.866242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 201.262300] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 201.338332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 203.744504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 203.748253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 203.756467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 203.760258] Lustre: Skipped 3 previous similar messages [ 208.176864] LustreError: 17877:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 208.180191] LustreError: 17877:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 208.241808] Lustre: server umount lustre-OST0000 complete [ 213.472235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 213.476104] LustreError: Skipped 1 previous similar message [ 213.478325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.485033] Lustre: Skipped 1 previous similar message [ 213.488212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 213.491721] Lustre: Skipped 2 previous similar messages [ 224.223152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 224.271277] LustreError: 18079:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 224.276025] LustreError: 18079:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 224.310956] Lustre: server umount lustre-MDT0000 complete [ 224.736590] LustreError: 15742:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 224.748914] LustreError: 15742:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 6 previous similar messages [ 225.795742] LustreError: 15726:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770446937 with bad export cookie 14055650397845298483 [ 225.800680] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 225.802257] LustreError: 15726:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 225.969301] Lustre: server umount lustre-MDT0001 complete [ 227.788788] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:48:58 (1770446938) [ 235.371957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.511815] LustreError: 19415:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.535634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.301046] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 240.610290] LustreError: 19415:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 241.341415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.061322] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 245.522969] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 246.831805] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 247.818755] Lustre: Failing over lustre-MDT0000 [ 247.920708] LustreError: 20755:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 247.923666] LustreError: 20755:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 247.952452] Lustre: server umount lustre-MDT0000 complete [ 250.849571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.850771] LustreError: 19414:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.859151] Lustre: Skipped 2 previous similar messages [ 256.993257] LustreError: 19414:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.005322] LustreError: 19414:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 260.741405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.775977] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.597721] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 266.037956] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 266.210166] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 266.218384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 266.222936] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 268.249109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.257742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.261392] Lustre: Skipped 5 previous similar messages [ 274.374743] LustreError: 21731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 274.376856] LustreError: 21731:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 274.407294] Lustre: server umount lustre-MDT0000 complete [ 276.226512] LustreError: 19395:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770446987 with bad export cookie 14055650397845299897 [ 276.233914] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.393858] Lustre: server umount lustre-MDT0001 complete [ 279.132218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.303269] LustreError: 22407:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.328621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.331298] Lustre: Skipped 2 previous similar messages [ 280.569630] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 283.758532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 285.727267] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 288.187280] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 289.588434] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 290.581391] Lustre: Failing over lustre-MDT0000 [ 290.685769] Lustre: server umount lustre-MDT0000 complete [ 293.666284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.674177] Lustre: Skipped 2 previous similar messages [ 298.976828] LustreError: 22406:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.986206] LustreError: 22406:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 305.010456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 305.075090] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.828068] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 309.344423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 309.347520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 309.349573] Lustre: Skipped 1 previous similar message [ 309.354157] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 310.172668] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 314.850571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 314.852547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.859407] Lustre: Skipped 3 previous similar messages [ 318.672778] LustreError: 24723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 318.677061] LustreError: 24723:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 318.711886] Lustre: server umount lustre-MDT0000 complete [ 320.546291] LustreError: 22388:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447032 with bad export cookie 14055650397845300828 [ 320.553078] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.553616] LustreError: 22388:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 320.699987] Lustre: server umount lustre-MDT0001 complete [ 324.069517] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:50:34 (1770447034) [ 331.040742] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 335.001442] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 352.273060] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 355.988899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.121641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 356.139112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 356.187089] Lustre: lustre-MDT0000: new disk, initializing [ 356.215345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.218880] Lustre: Skipped 2 previous similar messages [ 356.227313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 357.724787] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 361.632407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 361.668513] Lustre: 28379:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 361.766634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 362.913218] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 365.807094] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 366.978325] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 369.295589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 370.625769] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 370.649674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 370.657470] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 371.275613] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 374.204549] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 376.120968] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 380.896528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 380.898270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.898758] LustreError: Skipped 1 previous similar message [ 380.902239] Lustre: Skipped 2 previous similar messages [ 380.902899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 380.906335] Lustre: Skipped 1 previous similar message [ 385.531116] LustreError: 31007:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 385.533261] LustreError: 31007:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 385.577404] Lustre: server umount lustre-OST0000 complete [ 401.375091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 401.447498] Lustre: server umount lustre-MDT0000 complete [ 402.682248] LustreError: 28370:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447114 with bad export cookie 14055650397845301227 [ 402.687644] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 402.687752] LustreError: 28370:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 402.862669] Lustre: server umount lustre-MDT0001 complete [ 406.613251] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 407.536530] Key type lgssc unregistered [ 407.665414] LNet: 31943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.667929] LNetError: 31943:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 407.674369] LNet: Removed LNI 192.168.201.146@tcp [ 408.011097] Key type .llcrypt unregistered [ 408.012385] Key type ._llcrypt unregistered [ 416.732244] Key type ._llcrypt registered [ 416.733374] Key type .llcrypt registered [ 416.765568] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 417.070166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 417.099067] alg: No test for adler32 (adler32-zlib) [ 417.939191] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 418.013877] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 419.591149] Key type lgssc registered [ 420.112193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 422.824629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 422.830781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.987108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.292497] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 428.280334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 428.404207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.809302] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 431.784890] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 433.129941] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 435.779964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 435.931534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.995900] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 438.128616] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 439.875586] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.252682] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:52:45 (1770447165) [ 455.952903] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 456.639342] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 459.082396] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:52:49 (1770447169) [ 462.328428] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:52:52 (1770447172) [ 462.689962] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 465.515128] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 465.518476] Lustre: Skipped 2 previous similar messages [ 466.121984] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:52:56 (1770447176) [ 469.580612] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 469.584128] Lustre: Skipped 7 previous similar messages [ 474.252349] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:53:04 (1770447184) [ 475.185873] Lustre: *** cfs_fail_loc=131b, val=2*** [ 476.462748] Lustre: *** cfs_fail_loc=131b, val=2*** [ 480.065116] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:53:10 (1770447190) [ 480.770879] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 480.773334] Lustre: Skipped 6 previous similar messages [ 485.080132] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:53:15 (1770447195) [ 494.560452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.563813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.569899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.587594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 495.589803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.593456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.604720] Lustre: Skipped 1 previous similar message [ 500.704890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.708535] Lustre: Skipped 1 previous similar message [ 505.311640] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.346248] LustreError: 39860:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 505.404624] Lustre: server umount lustre-OST0000 complete [ 510.432103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.435797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.445710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.354349] LustreError: 40063:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 513.358020] LustreError: 40063:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 513.399359] Lustre: server umount lustre-MDT0000 complete [ 515.229937] LustreError: 33294:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447226 with bad export cookie 5979920599003584993 [ 515.235194] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.241631] LustreError: 33294:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 515.337316] LustreError: 40264:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 515.339683] LustreError: 40264:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 515.408622] Lustre: server umount lustre-MDT0001 complete [ 520.117575] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 521.352467] Key type lgssc unregistered [ 521.499460] LNet: 40795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.503694] LNetError: 40795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.517916] LNet: Removed LNI 192.168.201.146@tcp [ 521.894699] Key type .llcrypt unregistered [ 521.896509] Key type ._llcrypt unregistered [ 529.729110] Key type ._llcrypt registered [ 529.730511] Key type .llcrypt registered [ 529.770767] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 536.707567] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 537.013757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 537.055032] alg: No test for adler32 (adler32-zlib) [ 537.914776] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 537.999194] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 539.585637] Key type lgssc registered [ 540.035743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.545460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 552.551615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.642279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 553.654289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 553.693662] Lustre: lustre-MDT0000: new disk, initializing [ 553.717719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.727683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 555.158490] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 559.734418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 559.779411] Lustre: 43884:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 559.815742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 559.818166] Lustre: Skipped 1 previous similar message [ 559.854815] Lustre: lustre-MDT0001: new disk, initializing [ 559.888530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 559.905473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 559.909400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 561.304741] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 564.599698] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 565.996401] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.632379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 568.769875] Lustre: lustre-OST0000: new disk, initializing [ 568.772586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 568.812455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.008638] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 572.483842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 572.488862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 572.538112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 574.582600] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.059721] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 576.137071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 577.575324] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 577.657571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 582.623845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.625928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.631230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.137562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.138586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.140936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.365087] LustreError: 46817:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 584.408796] Lustre: server umount lustre-OST0000 complete [ 588.256715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.257434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.261429] Lustre: Skipped 1 previous similar message [ 588.265580] Lustre: Skipped 2 previous similar messages [ 593.376344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.379086] Lustre: Skipped 1 previous similar message [ 598.496240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 598.500896] Lustre: Skipped 1 previous similar message [ 600.031140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 600.067465] LustreError: 47018:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 600.069920] LustreError: 47018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 600.098940] Lustre: server umount lustre-MDT0000 complete [ 601.476429] LustreError: 43876:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447313 with bad export cookie 13536844521793093405 [ 601.479072] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.482648] LustreError: 43876:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 601.551555] LustreError: 47222:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 601.553761] LustreError: 47222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 601.610383] Lustre: server umount lustre-MDT0001 complete [ 605.174102] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:55:15 (1770447315) [ 607.653890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.794123] LustreError: 47940:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.799810] LustreError: 47940:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 607.815911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.216437] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 612.340232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 612.485253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 613.819465] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 615.541215] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 616.792956] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 619.052443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 621.106911] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 623.231661] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 636.048833] LustreError: 50156:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 636.399731] Lustre: Modifying parameter general.timeout=1 in log params [ 637.469053] Lustre: Modifying parameter general.timeout=4 in log params [ 637.471379] Lustre: Skipped 2 previous similar messages [ 639.612539] Lustre: Modifying parameter general.timeout=11 in log params [ 639.615112] Lustre: Skipped 6 previous similar messages [ 643.894450] Lustre: Modifying parameter general.timeout=25 in log params [ 643.896670] Lustre: Skipped 13 previous similar messages [ 651.966576] Lustre: Modifying parameter general.timeout=50 in log params [ 651.968498] Lustre: Skipped 24 previous similar messages [ 668.222112] Lustre: Modifying parameter general.timeout=100 in log params [ 668.225563] Lustre: Skipped 49 previous similar messages [ 683.401454] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:56:33 (1770447393) [ 684.705863] Lustre: 57052:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 686.324571] Lustre: 57249:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 686.331509] Lustre: 57249:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 688.837242] Lustre: 57559:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 688.841200] Lustre: 57559:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 700.192671] Lustre: 58900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 700.197734] LustreError: 58900:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 700.562960] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_policies= in log params [ 700.565614] Lustre: Skipped 50 previous similar messages [ 701.761044] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:56:51 (1770447411) [ 709.089586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.091077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.094017] Lustre: Skipped 2 previous similar messages [ 709.096293] Lustre: Skipped 1 previous similar message [ 720.351122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 720.401229] LustreError: 59595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 720.403970] LustreError: 59595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 720.447707] Lustre: server umount lustre-MDT0000 complete [ 722.082470] LustreError: 47964:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447433 with bad export cookie 13536844521793094294 [ 722.087184] LustreError: 47964:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 722.087443] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.236602] Lustre: server umount lustre-MDT0001 complete [ 734.199737] LustreError: 60001:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 734.202554] LustreError: 60001:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 734.251586] Lustre: server umount lustre-OST0000 complete [ 746.311394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.459620] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 746.590060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.592633] Lustre: Skipped 1 previous similar message [ 748.464042] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 752.507300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 752.574496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 752.587817] Lustre: 61426:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 754.561465] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 756.819912] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 758.466325] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 761.446234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 761.570308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 764.300506] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 766.685131] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 767.449504] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 767.454235] Lustre: Skipped 2 previous similar messages [ 773.089891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.093489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.099575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 773.103136] Lustre: Skipped 6 previous similar messages [ 776.178126] LustreError: 63895:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 776.182773] LustreError: 63895:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 776.235976] Lustre: server umount lustre-OST0000 complete [ 778.209496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 778.214068] LustreError: Skipped 1 previous similar message [ 784.242582] Lustre: server umount lustre-MDT0000 complete [ 785.902778] LustreError: 62151:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447497 with bad export cookie 13536844521793096639 [ 785.906407] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.907802] LustreError: 62151:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.071840] Lustre: server umount lustre-MDT0001 complete [ 791.062723] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 792.601690] Key type lgssc unregistered [ 792.803731] LNet: 64838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.809235] LNetError: 64838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.825545] LNet: Removed LNI 192.168.201.146@tcp [ 793.289752] Key type .llcrypt unregistered [ 793.292217] Key type ._llcrypt unregistered [ 805.788143] Key type ._llcrypt registered [ 805.789719] Key type .llcrypt registered [ 805.843675] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 806.212867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.231897] alg: No test for adler32 (adler32-zlib) [ 807.126854] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 807.251981] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 808.871253] Key type lgssc registered [ 809.401684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 812.995802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 813.012594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.344893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.183554] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 820.240152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 820.528947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 822.433840] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 824.819382] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 826.551490] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 829.749233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 830.002743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 832.836689] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 835.470814] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 849.932556] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 855.522653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.524510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.528505] Lustre: Skipped 2 previous similar messages [ 855.530869] Lustre: Skipped 1 previous similar message [ 859.139407] LustreError: 68931:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 859.168828] Lustre: server umount lustre-MDT0000 complete [ 860.641156] LustreError: 66203:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.649344] LustreError: 66203:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 860.745871] LustreError: 66187:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447572 with bad export cookie 14317389969191512818 [ 860.750287] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 860.756116] LustreError: 66187:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 860.831095] LustreError: 69136:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 860.834632] LustreError: 69136:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 860.908738] Lustre: server umount lustre-MDT0001 complete [ 872.884413] LustreError: 69336:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 872.886772] LustreError: 69336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 872.932636] Lustre: server umount lustre-OST0000 complete [ 883.968795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.077911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 884.091163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 884.208772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.751742] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 888.991072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 889.040552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 889.056902] Lustre: 70762:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 889.286042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 890.855070] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 892.930867] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 894.299276] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.982133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 897.084805] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 899.521568] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 901.781994] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 902.437218] Lustre: 72997:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 903.168891] Lustre: Modifying parameter general.jobid_var=disable in log params [ 908.258432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.258577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.261187] LustreError: Skipped 1 previous similar message [ 908.266549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.269729] Lustre: Skipped 2 previous similar messages [ 910.703077] LustreError: 73248:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 910.704771] LustreError: 73248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 910.755463] Lustre: server umount lustre-OST0000 complete [ 913.377069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.377977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.381114] Lustre: Skipped 2 previous similar messages [ 918.279742] LustreError: 73456:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 918.282569] LustreError: 73456:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 918.319828] Lustre: server umount lustre-MDT0000 complete [ 918.496137] LustreError: 71498:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.502794] LustreError: 71498:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 919.809789] LustreError: 70754:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447631 with bad export cookie 14317389969191513644 [ 919.812551] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.813816] LustreError: 70754:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 919.966630] Lustre: server umount lustre-MDT0001 complete [ 924.304663] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 925.471220] Key type lgssc unregistered [ 925.620344] LNet: 74188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.624532] LNetError: 74188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.632489] LNet: Removed LNI 192.168.201.146@tcp [ 925.950780] Key type .llcrypt unregistered [ 925.952046] Key type ._llcrypt unregistered [ 936.645199] Key type ._llcrypt registered [ 936.646640] Key type .llcrypt registered [ 936.694230] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 937.167452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.216832] alg: No test for adler32 (adler32-zlib) [ 938.121214] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 938.233513] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 939.831227] Key type lgssc registered [ 940.357887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.928422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.937540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.141896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.820612] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 950.703720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 951.016080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.851235] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 955.371634] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 956.873846] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 959.820518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 960.100759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.351368] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 964.475404] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 977.731692] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 978.634067] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 978.637555] Lustre: Skipped 1 previous similar message [ 979.961412] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 979.965667] Lustre: Skipped 2 previous similar messages [ 982.118417] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 982.121157] Lustre: Skipped 4 previous similar messages [ 986.378579] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 986.383283] Lustre: Skipped 10 previous similar messages [ 994.502291] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 994.504790] Lustre: Skipped 20 previous similar messages [ 1010.673990] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1010.676118] Lustre: Skipped 44 previous similar messages [ 1018.470752] Lustre: 83328:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1019.581799] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:02:09 (1770447729) [ 1020.233519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1020.932683] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:02:11 (1770447731) [ 1022.943954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.948280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.955342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1027.041791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.043393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.043602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.207090] LustreError: 83620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1028.255262] Lustre: server umount lustre-OST0000 complete [ 1032.160865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.168529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.171309] Lustre: Skipped 2 previous similar messages [ 1035.904678] LustreError: 83824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1035.907150] LustreError: 83824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1035.942092] Lustre: server umount lustre-MDT0000 complete [ 1037.280429] LustreError: 75557:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.289341] LustreError: 75557:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 1037.519706] LustreError: 75538:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447749 with bad export cookie 3470231099006566469 [ 1037.521530] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.523995] LustreError: 75538:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1037.608775] LustreError: 84025:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1037.611582] LustreError: 84025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1037.673234] Lustre: server umount lustre-MDT0001 complete [ 1041.941384] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1043.100923] Key type lgssc unregistered [ 1043.246273] LNet: 84555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.248957] LNetError: 84555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.256411] LNet: Removed LNI 192.168.201.146@tcp [ 1043.577211] Key type .llcrypt unregistered [ 1043.578805] Key type ._llcrypt unregistered [ 1045.018962] Key type ._llcrypt registered [ 1045.019849] Key type .llcrypt registered [ 1045.055434] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1046.681765] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules [ 1047.205978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1047.216525] alg: No test for adler32 (adler32-zlib) [ 1048.112985] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1048.216095] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1049.807155] Key type lgssc registered [ 1050.280809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1050.710469] LustreError: 85671:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1059.323143] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1062.943847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.876311] LustreError: 85671:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1071.887190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.122639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.674639] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1078.152827] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:03:08 (1770447788) [ 1080.128309] LustreError: 86708:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1080.131746] LustreError: 86708:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1080.135332] LustreError: 86708:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1080.212629] LustreError: 87479:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1080.280858] Lustre: server umount lustre-MDT0000 complete [ 1086.027747] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1087.207812] Key type lgssc unregistered [ 1087.356583] LNet: 88165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.358704] LNetError: 88165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.366336] LNet: Removed LNI 192.168.201.146@tcp [ 1087.696374] Key type .llcrypt unregistered [ 1087.697593] Key type ._llcrypt unregistered [ 1095.388304] Key type ._llcrypt registered [ 1095.389967] Key type .llcrypt registered [ 1095.432428] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1095.810509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.858050] alg: No test for adler32 (adler32-zlib) [ 1096.718974] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 1096.832215] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1098.423256] Key type lgssc registered [ 1098.873168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1102.238321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1102.246745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.452663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.096721] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1108.696123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1109.008875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1110.659413] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1112.673368] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.009038] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1116.635434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1116.934398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.247320] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1121.439220] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1154.238076] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:04:24 (1770447864) [ 1154.830227] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1155.475685] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:04:25 (1770447865) [ 1159.137099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.139193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.141298] Lustre: Skipped 2 previous similar messages [ 1159.144871] Lustre: Skipped 2 previous similar messages [ 1162.690764] LustreError: 92264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1162.725741] Lustre: server umount lustre-MDT0000 complete [ 1164.036463] LustreError: 90064:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447875 with bad export cookie 2101506701245572593 [ 1164.039912] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.041195] LustreError: 90064:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1164.117983] LustreError: 92467:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1164.122205] LustreError: 92467:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1164.174093] Lustre: server umount lustre-MDT0001 complete [ 1175.732904] LustreError: 92668:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1175.735594] LustreError: 92668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1175.850633] Lustre: server umount lustre-OST0000 complete [ 1180.465558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.655382] LustreError: 93296:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.668415] LustreError: 93296:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 1180.728656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.350087] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1185.625968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1185.889899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.414210] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1189.347973] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1190.719513] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1196.071524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1196.131053] LustreError: 95158:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.169693] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1196.174099] LustreError: 93284:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1196.186626] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1196.190390] LustreError: 95149:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -98 [ 1196.193464] LustreError: 95149:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1196.195813] LustreError: 95149:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1196.230362] Lustre: server umount lustre-OST0000 complete [ 1196.232995] LustreError: 95149:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1200.017440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1200.119431] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1200.123433] LustreError: 93284:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1200.135951] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1200.141638] LustreError: 95615:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -98 [ 1200.146846] LustreError: 95615:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1200.150871] LustreError: 95615:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1200.179906] Lustre: server umount lustre-OST0000 complete [ 1200.183116] LustreError: 95615:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1204.536887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1204.619144] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1204.622435] Lustre: Found index 0 for lustre-OST0000, updating log [ 1204.625857] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1204.661647] Lustre: lustre-OST0000: new disk, initializing [ 1204.663857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1204.779111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.120938] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1208.613139] Lustre: Failing over lustre-OST0000 [ 1208.639960] LustreError: 96683:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1208.643395] LustreError: 96683:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1208.684919] Lustre: server umount lustre-OST0000 complete [ 1210.848210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.848821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.854035] Lustre: Skipped 1 previous similar message [ 1215.968472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.972710] Lustre: Skipped 2 previous similar messages [ 1220.067682] LustreError: 96891:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1220.070709] LustreError: 96891:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1220.096070] Lustre: server umount lustre-MDT0000 complete [ 1221.089730] LustreError: 93295:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.098592] LustreError: 93295:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 1221.647964] LustreError: 93276:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770447933 with bad export cookie 2101506701245575323 [ 1221.651627] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.653898] LustreError: 93276:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1221.801022] Lustre: server umount lustre-MDT0001 complete [ 1225.070467] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:05:35 (1770447935) [ 1231.774574] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1236.207535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.378056] LustreError: 98473:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.414632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.028644] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1241.569827] LustreError: 98473:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.677644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1243.603144] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1244.831913] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1247.705381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1248.014499] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1250.423957] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1252.132807] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1252.136510] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 1252.141749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1254.017362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.054742] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1254.094835] Lustre: lustre-OST0001: new disk, initializing [ 1254.097737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1254.212637] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 1254.220917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1254.518939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1254.523939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1254.560125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1256.584417] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1257.546781] Lustre: lustre-OST0000: Denying connection for new client 86a045cd-f5cb-4a04-bbcd-b6d262b20333 (at 192.168.201.46@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 1260.004771] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 1260.011125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1260.339929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1261.041864] Lustre: lustre-OST0000: Recovery over after 0:09, of 1 clients 1 recovered and 0 were evicted. [ 1261.042290] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1261.051902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1266.425656] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1273.107038] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [ 1276.385589] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.393312] Lustre: Skipped 1 previous similar message [ 1278.215823] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [ 1278.219364] Lustre: Skipped 3 previous similar messages [ 1279.237864] LustreError: 101708:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1279.241031] LustreError: 101708:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1279.269159] Lustre: server umount lustre-MDT0001 complete [ 1281.509403] LustreError: 98467:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.517415] LustreError: 98467:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 1283.371870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1283.410873] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1283.413309] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1283.430380] Lustre: 98462:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1283.747284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1283.751862] Lustre: Skipped 3 previous similar messages [ 1284.903785] LustreError: 102391:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1284.908054] LustreError: 102391:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1284.912982] LustreError: 98464:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1284.964865] LustreError: 102412:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.146@tcp: cfg command failed: rc = -17 [ 1284.970409] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1284.970409] [ 1285.433080] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1289.191864] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1289.952185] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 02:06:40 (1770448000) [ 1297.334886] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1304.408384] Lustre: 104307:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1314.218597] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1315.735972] Lustre: 105439:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1370.592042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.596793] LustreError: Skipped 1 previous similar message [ 1370.598407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.602830] Lustre: Skipped 1 previous similar message [ 1370.604469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.771469] LustreError: 105893:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1375.775383] LustreError: 105893:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1375.807538] Lustre: server umount lustre-MDT0000 complete [ 1376.224183] LustreError: 99950:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.232543] LustreError: 99950:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 1377.184883] LustreError: 99578:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448088 with bad export cookie 2101506701245576310 [ 1377.186252] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.188866] LustreError: 99578:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1377.327642] Lustre: server umount lustre-MDT0001 complete [ 1388.557996] Lustre: server umount lustre-OST0000 complete [ 1400.238857] Lustre: server umount lustre-OST0001 complete [ 1402.408865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1406.332161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1409.631344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1413.446074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1421.080985] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1424.893117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.899497] Lustre: lustre-MDT0000: reset Object Index mappings [ 1425.086376] LustreError: 109365:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.118108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.586863] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1429.929647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1429.937779] Lustre: lustre-MDT0001: reset Object Index mappings [ 1431.837945] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1432.998939] Lustre: 110476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1435.766079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1438.318307] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1440.236323] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1441.817081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1443.189614] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1444.473971] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1450.347802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1450.347888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 1453.467456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1454.974561] Lustre: 112325:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1477.058414] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:09:47 (1770448187) [ 1506.786575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.789624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.791835] Lustre: Skipped 4 previous similar messages [ 1506.792872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1506.795337] Lustre: Skipped 6 previous similar messages [ 1512.277615] LustreError: 112675:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1512.282594] LustreError: 112675:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1512.325440] Lustre: server umount lustre-MDT0000 complete [ 1513.962462] LustreError: 110096:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448225 with bad export cookie 2101506701245623294 [ 1513.963080] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.967401] LustreError: 110096:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1517.024975] LustreError: 110854:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.025872] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.032373] LustreError: 110854:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 1517.038186] Lustre: Skipped 2 previous similar messages [ 1520.410837] Lustre: server umount lustre-MDT0001 complete [ 1528.271899] Lustre: server umount lustre-OST0000 complete [ 1544.159151] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1544.247396] Lustre: server umount lustre-OST0001 complete [ 1546.699070] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 1550.693640] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1564.828979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.966872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1564.982988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1565.037993] Lustre: lustre-MDT0000: new disk, initializing [ 1565.069128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.071653] Lustre: Skipped 3 previous similar messages [ 1565.079746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1566.657854] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1569.982814] Lustre: server umount lustre-MDT0000 complete [ 1573.763501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.895323] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1573.902119] Lustre: 116533:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1573.910236] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1575.520477] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1577.414867] LustreError: 117047:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1577.417943] LustreError: 117047:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 1577.474907] Lustre: server umount lustre-MDT0000 complete [ 1580.766830] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:11:30 (1770448290) [ 1581.367430] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1582.094857] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:11:32 (1770448292) [ 1588.522916] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1592.733456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.386044] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1597.441700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1597.494421] Lustre: 118549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1597.530184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1597.532624] Lustre: Skipped 1 previous similar message [ 1597.572887] Lustre: lustre-MDT0001: new disk, initializing [ 1597.629170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1597.633974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1599.019182] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1601.406516] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1604.002435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1604.109553] Lustre: lustre-OST0000: new disk, initializing [ 1604.111459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1605.312867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1605.318228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1605.372582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1606.217643] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1610.787293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1612.171603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1612.175705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1612.208681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1613.090990] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1617.891334] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1619.219831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1776.718085] Lustre: 127306:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1776.722066] Lustre: 127306:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1776.724177] Lustre: 127306:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1776.726259] Lustre: 127306:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1776.729751] Lustre: 127306:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1776.731750] Lustre: 127306:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1857.491564] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:16:07 (1770448567) [ 1873.378281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.393154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.398026] Lustre: Skipped 3 previous similar messages [ 1873.411632] Lustre: Skipped 11 previous similar messages [ 1878.789524] LustreError: 127511:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1878.819437] LustreError: 127511:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1879.946254] Lustre: server umount lustre-MDT0000 complete [ 1883.949191] LustreError: 118554:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.956973] LustreError: 118554:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 1883.976107] hrtimer: interrupt took 4099604 ns [ 1889.242946] LustreError: 121691:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770448600 with bad export cookie 2101506701245678027 [ 1889.258388] LustreError: 121691:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1889.260794] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1895.814041] Lustre: server umount lustre-MDT0001 complete [ 1907.075664] Lustre: server umount lustre-OST0000 complete [ 1920.951213] Lustre: server umount lustre-OST0001 complete [ 1935.099745] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 1952.937249] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1999.459064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.865507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1999.916361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1999.929462] Lustre: Skipped 1 previous similar message [ 2000.042427] Lustre: lustre-MDT0000: new disk, initializing [ 2000.046412] Lustre: Skipped 1 previous similar message [ 2000.149875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.157199] Lustre: Skipped 5 previous similar messages [ 2000.170988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2004.931852] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2017.087583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2017.216451] Lustre: 130321:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2017.274607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2017.278086] Lustre: Skipped 1 previous similar message [ 2017.384879] Lustre: lustre-MDT0001: new disk, initializing [ 2017.542344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2017.565789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2021.618356] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2029.189183] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2033.323475] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2040.436894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2042.264716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2042.420312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2047.502315] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2055.203598] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2064.750690] Lustre: lustre-MDD0000: changelog on [ 2067.427843] Lustre: lustre-MDD0001: changelog on [ 4060.614533] Lustre: lustre-MDD0001: changelog off [ 4062.251174] Lustre: 132410:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4063.047418] Lustre: lustre-MDD0000: changelog off [ 4069.869143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.876347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.894579] Lustre: Skipped 4 previous similar messages [ 4069.900442] Lustre: Skipped 8 previous similar messages [ 4069.915208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4074.060884] LustreError: 140166:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4074.072545] LustreError: 140166:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4074.247852] Lustre: server umount lustre-OST0000 complete [ 4080.097616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.102931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4080.113432] Lustre: Skipped 1 previous similar message [ 4080.132785] Lustre: Skipped 1 previous similar message [ 4091.871128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4092.051062] Lustre: server umount lustre-MDT0000 complete [ 4095.257723] LustreError: 130313:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450806 with bad export cookie 2101506701246445094 [ 4095.269046] LustreError: 130313:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4095.269853] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.493574] Lustre: server umount lustre-MDT0001 complete [ 4102.633609] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4104.518158] Key type lgssc unregistered [ 4104.722023] LNet: 141104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4104.727993] LNetError: 141104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4104.742982] LNet: Removed LNI 192.168.201.146@tcp [ 4105.341331] Key type .llcrypt unregistered [ 4105.345263] Key type ._llcrypt unregistered [ 4118.228795] Key type ._llcrypt registered [ 4118.232722] Key type .llcrypt registered [ 4118.318655] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4129.535545] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4130.323243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4130.347171] alg: No test for adler32 (adler32-zlib) [ 4131.298938] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4131.448987] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4133.071729] Key type lgssc registered [ 4133.730648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4157.325037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4157.339686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.526976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4158.561085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4158.620723] Lustre: lustre-MDT0000: new disk, initializing [ 4158.679773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.697932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4161.178468] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4168.466035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4168.573058] Lustre: 144242:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4168.647648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4168.653125] Lustre: Skipped 1 previous similar message [ 4168.726421] Lustre: lustre-MDT0001: new disk, initializing [ 4168.800504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4168.829411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4168.838583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4171.707094] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4177.157718] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4179.841244] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4184.269598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4184.465535] Lustre: lustre-OST0000: new disk, initializing [ 4184.469089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4184.546066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4188.135709] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4193.093656] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4193.815917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4193.827243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4193.905978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4197.645401] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_module obdecho/obdecho [ 4198.829198] LustreError: 146858:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4198.835588] LustreError: 146858:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4200.126185] LustreError: 146954:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4207.013510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.017423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.023219] Lustre: Skipped 2 previous similar messages [ 4207.030784] Lustre: Skipped 2 previous similar messages [ 4212.192986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.196330] Lustre: Skipped 2 previous similar messages [ 4212.753600] LustreError: 147294:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4212.800525] Lustre: server umount lustre-MDT0000 complete [ 4215.041327] LustreError: 144234:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770450926 with bad export cookie 9746837359103457345 [ 4215.047504] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.047703] LustreError: 144234:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.143658] LustreError: 147496:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4215.148234] LustreError: 147496:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4215.253812] Lustre: server umount lustre-MDT0001 complete [ 4227.789477] LustreError: 147696:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4227.798307] LustreError: 147696:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4227.877389] Lustre: server umount lustre-OST0000 complete [ 4241.433968] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4249.113685] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4272.707149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.833566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4272.857062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4272.915279] Lustre: lustre-MDT0000: new disk, initializing [ 4272.996132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.007533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4275.206276] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4282.325631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4282.413064] Lustre: 150628:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4282.471333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4282.475668] Lustre: Skipped 1 previous similar message [ 4282.544987] Lustre: lustre-MDT0001: new disk, initializing [ 4282.628339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.653325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4282.661250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4284.838154] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4289.818423] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4291.823198] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4296.334260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4296.534107] Lustre: lustre-OST0000: new disk, initializing [ 4296.537064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4296.591772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.084574] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4301.869085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4301.877552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4301.944283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4305.194545] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4307.493835] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4307.621079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4309.653457] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4309.751734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4312.034087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.035399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.038214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.045031] Lustre: Skipped 1 previous similar message [ 4316.720173] LustreError: 153560:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4316.724026] LustreError: 153560:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4316.784135] Lustre: server umount lustre-OST0000 complete [ 4318.688446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.695783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.703624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.707865] Lustre: Skipped 1 previous similar message [ 4322.276655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.691599] LustreError: 153762:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4324.695487] LustreError: 153762:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4324.753555] Lustre: server umount lustre-MDT0000 complete [ 4326.906608] LustreError: 150621:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451038 with bad export cookie 9746837359103458304 [ 4326.912057] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.919395] LustreError: 150621:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4327.126413] Lustre: server umount lustre-MDT0001 complete [ 4329.249079] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:57:19 (1770451039) [ 4330.157225] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4331.157667] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:57:21 (1770451041) [ 4336.315597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.728872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.234491] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4341.728190] LustreError: 154635:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.737729] LustreError: 154635:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 4344.566662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4347.701196] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4351.546893] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4354.117329] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4358.511797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4362.167044] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4365.679948] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4375.524076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4375.526106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.530462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4375.536672] Lustre: Skipped 2 previous similar messages [ 4375.540076] Lustre: Skipped 3 previous similar messages [ 4377.189523] LustreError: 156878:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4377.203428] LustreError: 156878:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4377.275715] Lustre: server umount lustre-MDT0000 complete [ 4379.290528] LustreError: 156248:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451090 with bad export cookie 9746837359103459186 [ 4379.296221] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.301884] LustreError: 156248:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4380.640314] LustreError: 155790:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.672789] LustreError: 155790:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.673267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4384.684120] LustreError: 155790:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 4384.690115] Lustre: Skipped 4 previous similar messages [ 4385.657188] Lustre: server umount lustre-MDT0001 complete [ 4389.219058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4397.312121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4397.491753] LustreError: 158136:0:(llog.c:1643:llog_backup()) MGC192.168.201.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4397.498586] Lustre: 158136:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4401.992194] LustreError: 158143:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.110037] LustreError: 158143:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.121329] LustreError: 158143:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 4418.310183] LustreError: 158142:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.320457] LustreError: 158142:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 8 previous similar messages [ 4437.765163] LustreError: 158142:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.773761] LustreError: 158142:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 16 previous similar messages [ 4458.975466] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4458.984355] Lustre: 158136:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4472.586176] LustreError: 158161:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.598378] LustreError: 158161:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 20 previous similar messages [ 4488.671566] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4488.677115] Lustre: 154626:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4499.503544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4499.511235] Lustre: Skipped 2 previous similar messages [ 4501.745726] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4503.525258] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4506.477130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.151312] LustreError: 154626:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4524.000098] LustreError: 142467:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f3485fb6 x1856452262210304/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4524.135958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 4526.356375] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4529.302393] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4529.639831] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4530.323837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4532.580364] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4534.730661] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4544.999120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.001322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4545.002333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4545.015383] Lustre: Skipped 3 previous similar messages [ 4546.104357] LustreError: 159622:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4546.107476] LustreError: 159622:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4546.166424] Lustre: server umount lustre-MDT0000 complete [ 4548.189319] LustreError: 154615:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451259 with bad export cookie 9746837359103465654 [ 4548.192319] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.195806] LustreError: 154615:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4548.381182] Lustre: server umount lustre-MDT0001 complete [ 4560.886412] Lustre: server umount lustre-OST0000 complete [ 4572.346146] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4578.251370] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4600.476068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.622262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4600.654610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4600.734979] Lustre: lustre-MDT0000: new disk, initializing [ 4600.797702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.802414] Lustre: Skipped 1 previous similar message [ 4600.811901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4603.405834] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4610.019556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4610.100431] Lustre: 162966:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4610.348546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4612.855288] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4617.357526] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4619.461269] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4623.755350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4627.695221] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4631.073147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4631.152677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4632.774982] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4635.077686] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4635.199715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4637.584859] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4637.761713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4641.258378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.259776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4641.261414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4641.261421] Lustre: Skipped 2 previous similar messages [ 4641.265296] Lustre: Skipped 1 previous similar message [ 4645.042760] LustreError: 165895:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4645.048080] LustreError: 165895:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4645.111421] Lustre: server umount lustre-OST0000 complete [ 4653.316195] Lustre: server umount lustre-MDT0000 complete [ 4655.533516] LustreError: 162960:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451367 with bad export cookie 9746837359103484260 [ 4655.535955] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.541216] LustreError: 162960:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4655.729604] Lustre: server umount lustre-MDT0001 complete [ 4661.490236] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:02:51 (1770451371) [ 4665.340440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.663243] LustreError: 166968:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.676907] LustreError: 166968:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 40 previous similar messages [ 4668.103867] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4672.980452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4675.458819] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4678.373612] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4680.153080] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4683.984746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4687.447090] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4690.551366] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4706.028256] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:03:35 (1770451415) [ 4707.815858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.818010] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4707.819320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4707.819325] Lustre: Skipped 3 previous similar messages [ 4707.832134] Lustre: Skipped 3 previous similar messages [ 4707.837869] LustreError: Skipped 1 previous similar message [ 4713.842720] LustreError: 169428:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4713.846550] LustreError: 169428:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4713.921392] Lustre: server umount lustre-OST0000 complete [ 4722.297938] Lustre: server umount lustre-MDT0000 complete [ 4724.428869] LustreError: 166950:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451435 with bad export cookie 9746837359103485142 [ 4724.433499] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4724.435215] LustreError: 166950:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4724.605758] Lustre: server umount lustre-MDT0001 complete [ 4730.585305] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4732.292679] Key type lgssc unregistered [ 4732.487433] LNet: 170362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4732.494599] LNetError: 170362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4732.509862] LNet: Removed LNI 192.168.201.146@tcp [ 4733.006136] Key type .llcrypt unregistered [ 4733.008392] Key type ._llcrypt unregistered [ 4744.668232] Key type ._llcrypt registered [ 4744.670272] Key type .llcrypt registered [ 4744.738224] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4745.264095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4745.329836] alg: No test for adler32 (adler32-zlib) [ 4746.256682] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4746.397546] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4748.007180] Key type lgssc registered [ 4748.646334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4753.380044] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4753.392500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4768.929777] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4774.047199] LustreError: 171562:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 4779.167384] LustreError: 171562:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4779.175205] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4779.178874] LustreError: MGC192.168.201.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4779.185312] LustreError: 171562:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4779.191196] LustreError: 171562:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -22 [ 4779.195553] LustreError: 171562:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4779.201477] LustreError: 171562:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4779.236856] Lustre: server umount lustre-OST0000 complete [ 4779.239605] LustreError: 171562:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4784.286652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.676470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.428863] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4793.508486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4793.796847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4796.344492] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4799.907590] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4802.545135] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4806.823625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4806.984880] LustreError: 173650:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4806.991476] Lustre: 173650:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4807.090472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.798360] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4814.688286] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4818.597703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.601969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4818.627955] Lustre: Skipped 2 previous similar messages [ 4818.636276] Lustre: Skipped 1 previous similar message [ 4822.701115] LustreError: 174301:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4822.704383] LustreError: 174301:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4822.772300] Lustre: server umount lustre-MDT0000 complete [ 4824.034117] LustreError: 172062:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4824.051533] LustreError: 172062:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 4 previous similar messages [ 4825.410980] LustreError: 172040:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451536 with bad export cookie 11034165956737333836 [ 4825.416900] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.425472] LustreError: 172040:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4826.847241] LustreError: 172774:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005658b817 x1856452907000064/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4826.873379] LustreError: 174502:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4826.880579] LustreError: 174502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4826.971854] Lustre: server umount lustre-MDT0001 complete [ 4840.063132] LustreError: 174703:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4840.067704] LustreError: 174703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4840.169405] Lustre: server umount lustre-OST0000 complete [ 4845.994524] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4852.058982] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4877.578470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.786644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4877.813355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4877.882752] Lustre: lustre-MDT0000: new disk, initializing [ 4877.970952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.985452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4881.137400] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4889.735135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4889.819468] Lustre: 177027:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4889.858719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4889.862666] Lustre: Skipped 1 previous similar message [ 4889.920416] Lustre: lustre-MDT0001: new disk, initializing [ 4890.024673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4890.063812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4890.083552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4892.721551] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4898.317677] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4900.808250] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4905.086728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4905.296865] Lustre: lustre-OST0000: new disk, initializing [ 4905.301381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4905.367746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.193895] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4912.689096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4912.697573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4912.786265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4914.670610] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4917.811878] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4917.958915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4920.283936] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4920.436395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4922.848248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4922.854184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.863295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4923.365726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4927.608119] LustreError: 179960:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4927.612265] LustreError: 179960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4927.685457] Lustre: server umount lustre-OST0000 complete [ 4931.040584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4931.046384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.053281] Lustre: Skipped 1 previous similar message [ 4931.057467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.064017] Lustre: Skipped 2 previous similar messages [ 4933.606412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.607034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.721288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4938.727070] Lustre: Skipped 2 previous similar messages [ 4944.351151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4944.427331] LustreError: 180161:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4944.435127] LustreError: 180161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4944.491687] Lustre: server umount lustre-MDT0000 complete [ 4946.880249] LustreError: 177018:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451658 with bad export cookie 11034165956737334557 [ 4946.885496] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4946.887340] LustreError: 177018:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4947.092404] Lustre: server umount lustre-MDT0001 complete [ 4952.757966] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:07:42 (1770451662) [ 4962.360372] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4963.868316] Key type lgssc unregistered [ 4964.070685] LNet: 181547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4964.077137] LNetError: 181547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4964.086549] LNet: Removed LNI 192.168.201.146@tcp [ 4964.606090] Key type .llcrypt unregistered [ 4964.608428] Key type ._llcrypt unregistered [ 4975.177867] Key type ._llcrypt registered [ 4975.180554] Key type .llcrypt registered [ 4975.246447] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4975.845133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4975.855382] alg: No test for adler32 (adler32-zlib) [ 4976.793383] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 4976.936031] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4978.543485] Key type lgssc registered [ 4979.170960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4983.227564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4983.240930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.529744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4986.882461] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4991.576809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4991.769976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4993.775263] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4996.873963] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4999.089953] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5003.396400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5003.521500] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5003.524875] LustreError: 184357:0:(llog.c:1684:llog_backup()) MGC192.168.201.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5003.532397] Lustre: 184357:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5003.610545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.725392] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5009.786875] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5012.448658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.453858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.463284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.010539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5015.010873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.020023] Lustre: Skipped 1 previous similar message [ 5015.025106] Lustre: Skipped 1 previous similar message [ 5017.057585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5017.062538] Lustre: Skipped 2 previous similar messages [ 5022.177375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.181508] Lustre: Skipped 2 previous similar messages [ 5025.247097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5025.292523] LustreError: 185007:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5025.322468] Lustre: server umount lustre-MDT0000 complete [ 5027.084806] LustreError: 182702:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451738 with bad export cookie 18101425572779150981 [ 5027.089440] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.093308] LustreError: 182702:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5027.255239] LustreError: 183433:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b23c4009 x1856453149219968/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5027.270341] LustreError: 185211:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5027.274070] LustreError: 185211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5027.296226] LustreError: 182718:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.296984] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.306583] LustreError: 182718:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5027.356229] Lustre: server umount lustre-MDT0001 complete [ 5035.566584] LustreError: 185413:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5035.571927] LustreError: 185413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5035.638743] Lustre: server umount lustre-OST0000 complete [ 5040.689560] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5046.546316] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5066.633368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.818095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5066.844341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5066.907992] Lustre: lustre-MDT0000: new disk, initializing [ 5066.979503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.999888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5069.410284] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5075.971123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5076.037161] Lustre: 187737:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5076.072567] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5076.075449] Lustre: Skipped 1 previous similar message [ 5076.078317] LustreError: 188500:0:(llog.c:1684:llog_backup()) MGC192.168.201.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5076.083286] LustreError: 188500:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5076.089218] Lustre: 188500:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5076.103400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5076.107605] Lustre: Skipped 1 previous similar message [ 5076.169493] Lustre: lustre-MDT0001: new disk, initializing [ 5076.239187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5076.266293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5076.273037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5078.331971] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5082.627641] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5084.709000] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5088.415401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5088.568939] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5088.571752] Lustre: Skipped 1 previous similar message [ 5088.575059] LustreError: 189453:0:(llog.c:1684:llog_backup()) MGC192.168.201.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5088.580993] LustreError: 189453:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5088.585451] Lustre: 189453:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5088.591587] Lustre: 189453:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5088.605799] Lustre: lustre-OST0000: new disk, initializing [ 5088.609428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5088.671361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.969081] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5092.557221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5092.564699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5092.630447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5096.707986] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5099.159052] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5099.272612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5101.696087] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5101.812672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5103.078606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.078989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.083680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5103.104807] Lustre: Skipped 1 previous similar message [ 5108.847124] LustreError: 190666:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5108.852355] LustreError: 190666:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5108.923112] Lustre: server umount lustre-OST0000 complete [ 5112.289229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5112.294100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.304153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5112.306815] Lustre: Skipped 3 previous similar messages [ 5117.198492] LustreError: 190867:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5117.204307] LustreError: 190867:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5117.251842] Lustre: server umount lustre-MDT0000 complete [ 5118.432398] LustreError: 187744:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.442619] LustreError: 187744:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5119.386974] LustreError: 188512:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451830 with bad export cookie 18101425572779151688 [ 5119.390508] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5119.393951] LustreError: 188512:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5119.589942] Lustre: server umount lustre-MDT0001 complete [ 5124.569298] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:10:34 (1770451834) [ 5133.365250] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5138.669460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.903768] LustreError: 192435:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.927410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.069510] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5144.032574] LustreError: 192435:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5145.208464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5147.349242] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5148.862945] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5152.133910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5155.291771] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5159.632395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5159.725709] Lustre: lustre-OST0001: new disk, initializing [ 5159.728224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5160.158808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5160.163087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5160.223293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5162.568213] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5169.123520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5175.644876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5181.585964] Lustre: DEBUG MARKER: ADD OST3 [ 5184.011405] loop: module loaded [ 5184.036686] loop0: detected capacity change from 0 to 204800000 [ 5184.183895] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5184.190685] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5184.197448] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5187.156633] loop0: detected capacity change from 0 to 204800000 [ 5189.586933] Lustre: DEBUG MARKER: STOP OST3 [ 5189.983821] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5190.098568] Lustre: lustre-OST0003: new disk, initializing [ 5190.103483] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5190.180289] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5190.186092] Lustre: Skipped 3 previous similar messages [ 5191.293688] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5191.356926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5191.363374] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5191.384333] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5192.481210] Lustre: Failing over lustre-OST0003 [ 5192.528189] LustreError: 196930:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5192.533313] LustreError: 196930:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5192.559151] Lustre: server umount lustre-OST0003 complete [ 5193.223216] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5193.312290] LustreError: 196735:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5196.434576] LustreError: 193885:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.774676] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.780241] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5196.784336] Lustre: Skipped 2 previous similar messages [ 5198.316258] Lustre: DEBUG MARKER: START OST3 again [ 5201.431036] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5201.584136] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5201.672029] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5203.392664] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5203.392695] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5204.366301] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5220.092138] Lustre: Failing over lustre-OST0003 [ 5220.141458] LustreError: 198862:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5220.146917] LustreError: 198862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5220.181364] Lustre: server umount lustre-OST0003 complete [ 5222.379559] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.384028] LustreError: 194246:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.387281] Lustre: Skipped 1 previous similar message [ 5222.396572] LustreError: 194246:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5237.729347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5237.735478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5237.738414] Lustre: Skipped 5 previous similar messages [ 5240.320162] Lustre: server umount lustre-MDT0000 complete [ 5242.180573] LustreError: 192414:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770451953 with bad export cookie 18101425572779152570 [ 5242.183503] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5242.187916] LustreError: 192414:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5242.848566] LustreError: 192429:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.862638] LustreError: 192429:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 7 previous similar messages [ 5248.477672] Lustre: server umount lustre-MDT0001 complete [ 5250.757518] Lustre: server umount lustre-OST0000 complete [ 5252.722333] LustreError: 199738:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5252.726358] LustreError: 199738:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 5252.800142] Lustre: server umount lustre-OST0001 complete [ 5256.030560] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5260.706807] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5280.076268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.207551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5280.225141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5280.277689] Lustre: lustre-MDT0000: new disk, initializing [ 5280.318414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.322145] Lustre: Skipped 1 previous similar message [ 5280.331822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5282.311968] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5288.346547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5288.414758] Lustre: 201910:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5288.592702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5290.671729] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5294.998244] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.749734] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5300.079553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5300.247123] Lustre: lustre-OST0000: new disk, initializing [ 5300.250602] Lustre: Skipped 1 previous similar message [ 5300.255407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5300.258337] Lustre: Skipped 2 previous similar messages [ 5301.777320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5301.782463] Lustre: Skipped 1 previous similar message [ 5301.785532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5301.847412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5303.142188] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5307.229836] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5309.148523] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5309.255805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5311.109846] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5311.210699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5317.091075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.093620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.095714] LustreError: Skipped 1 previous similar message [ 5317.100678] Lustre: Skipped 6 previous similar messages [ 5317.103636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.109506] Lustre: Skipped 5 previous similar messages [ 5318.193295] LustreError: 204839:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5318.196526] LustreError: 204839:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5318.250539] Lustre: server umount lustre-OST0000 complete [ 5326.185715] Lustre: server umount lustre-MDT0000 complete [ 5327.329068] LustreError: 202692:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.336613] LustreError: 202692:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 5327.891142] LustreError: 201901:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452039 with bad export cookie 18101425572779154859 [ 5327.893891] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5327.897397] LustreError: 201901:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5328.054744] Lustre: server umount lustre-MDT0001 complete [ 5329.837145] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:13:59 (1770452039) [ 5337.417938] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5341.879396] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5357.456253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.581951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5357.599838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5357.649333] Lustre: lustre-MDT0000: new disk, initializing [ 5357.704446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5359.468707] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5364.749114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5364.810978] Lustre: 208070:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5364.994636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5366.936913] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5371.238209] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5372.975961] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5376.463305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5378.548957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5379.294255] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5382.961036] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5384.843557] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5384.950689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5386.880597] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5386.994898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5388.770846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5388.774685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.776505] LustreError: Skipped 1 previous similar message [ 5388.777664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5388.783587] Lustre: Skipped 3 previous similar messages [ 5388.791910] Lustre: Skipped 3 previous similar messages [ 5394.088168] Lustre: server umount lustre-OST0000 complete [ 5410.271116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5410.346182] Lustre: server umount lustre-MDT0000 complete [ 5411.928618] LustreError: 208849:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452123 with bad export cookie 18101425572779155741 [ 5411.933466] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.934803] LustreError: 208849:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5412.091592] Lustre: server umount lustre-MDT0001 complete [ 5415.709367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.914689] LustreError: 211883:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.955230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.958968] Lustre: Skipped 5 previous similar messages [ 5417.691743] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5421.241401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5423.148525] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5425.507464] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5427.182840] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5430.485280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5433.306464] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5435.988745] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5451.750472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5451.753539] LustreError: Skipped 1 previous similar message [ 5454.448457] LustreError: 214242:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5454.451298] LustreError: 214242:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 5454.503113] Lustre: server umount lustre-OST0000 complete [ 5462.396142] Lustre: server umount lustre-MDT0000 complete [ 5464.238961] LustreError: 214143:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452175 with bad export cookie 18101425572779156623 [ 5464.241687] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.245171] LustreError: 214143:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5464.416328] Lustre: server umount lustre-MDT0001 complete [ 5572.028958] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5573.373827] Key type lgssc unregistered [ 5573.528707] LNet: 215213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5573.533599] LNetError: 215213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5573.546883] LNet: Removed LNI 192.168.201.146@tcp [ 5573.985114] Key type .llcrypt unregistered [ 5573.987802] Key type ._llcrypt unregistered [ 5585.445723] Key type ._llcrypt registered [ 5585.447657] Key type .llcrypt registered [ 5585.501514] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5592.744971] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5593.375250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5593.403221] alg: No test for adler32 (adler32-zlib) [ 5594.306441] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 5594.406165] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5595.999172] Key type lgssc registered [ 5596.448122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5613.435746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5613.446617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.592885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5614.614364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5614.671542] Lustre: lustre-MDT0000: new disk, initializing [ 5614.742602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.753951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5616.944794] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5622.554349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5622.620323] Lustre: 218502:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.657834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5622.660995] Lustre: Skipped 1 previous similar message [ 5622.710762] Lustre: lustre-MDT0001: new disk, initializing [ 5622.761446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5622.788417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5622.796917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5624.680774] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5628.731272] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5630.452422] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5633.627052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5633.765758] Lustre: lustre-OST0000: new disk, initializing [ 5633.769357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5633.810129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.038921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5636.043713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5636.090853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5636.406671] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5640.273239] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5642.293806] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5642.399093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5644.107043] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5644.193389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5646.305281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.307434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.311662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.317543] Lustre: Skipped 1 previous similar message [ 5650.995083] LustreError: 221434:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5651.039690] Lustre: server umount lustre-OST0000 complete [ 5653.472463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5653.477022] LustreError: Skipped 1 previous similar message [ 5653.479512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.486473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.490354] Lustre: Skipped 1 previous similar message [ 5656.544532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.544982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.824356] LustreError: 221635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5658.827750] LustreError: 221635:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5658.859879] Lustre: server umount lustre-MDT0000 complete [ 5660.408326] LustreError: 218493:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452371 with bad export cookie 14560473634327982392 [ 5660.412106] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.414374] LustreError: 218493:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5660.498227] LustreError: 221836:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5660.502204] LustreError: 221836:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5660.571404] Lustre: server umount lustre-MDT0001 complete [ 5668.860849] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5673.657293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.869270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.592936] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5679.072488] LustreError: 223008:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.079374] LustreError: 223008:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5679.223034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5679.365857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5681.170805] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5682.463992] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5685.347508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5687.872034] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5691.710566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.797502] Lustre: lustre-OST0001: new disk, initializing [ 5691.800416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5691.838818] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5691.842573] Lustre: Skipped 1 previous similar message [ 5693.265547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5693.272305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5693.321716] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5694.237761] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5700.571970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5706.027267] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5716.379576] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:20:26 (1770452426) [ 5719.498629] LustreError: 226379:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5719.501869] LustreError: 226379:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5719.544849] Lustre: server umount lustre-MDT0000 complete [ 5720.034991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.039392] LustreError: 223002:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.045588] Lustre: Skipped 2 previous similar messages [ 5720.054763] LustreError: 223002:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 5721.270886] LustreError: 224107:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452432 with bad export cookie 14560473634327983274 [ 5721.275390] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5721.277374] LustreError: 224107:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5721.448306] Lustre: server umount lustre-MDT0001 complete [ 5733.371115] LustreError: 226780:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5733.376202] LustreError: 226780:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5733.397627] Lustre: server umount lustre-OST0000 complete [ 5745.460722] LustreError: 226982:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5745.464222] LustreError: 226982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5745.523584] Lustre: server umount lustre-OST0001 complete [ 5748.424322] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5753.502283] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5770.926079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.057689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5771.077629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5771.131673] Lustre: lustre-MDT0000: new disk, initializing [ 5771.168106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.177887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5773.048341] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5778.172950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5778.236978] Lustre: 229153:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5778.387350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5780.323313] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5784.519786] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5786.256608] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5789.315996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5789.443931] Lustre: lustre-OST0000: new disk, initializing [ 5789.446305] Lustre: Skipped 1 previous similar message [ 5789.449221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5789.451488] Lustre: Skipped 2 previous similar messages [ 5790.727504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5790.732680] Lustre: Skipped 1 previous similar message [ 5790.736834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5790.804361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5792.441477] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5796.559599] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5798.577379] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5798.687830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5800.757486] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5800.874110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5806.052963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.053528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5806.055206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5806.055210] Lustre: Skipped 1 previous similar message [ 5806.062441] Lustre: Skipped 1 previous similar message [ 5811.170905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5811.173901] Lustre: Skipped 2 previous similar messages [ 5816.287146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5816.310326] LustreError: 232080:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5816.314308] LustreError: 232080:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5816.375838] Lustre: server umount lustre-OST0000 complete [ 5819.360497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.365949] LustreError: Skipped 1 previous similar message [ 5819.369018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.377788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5819.381885] Lustre: Skipped 2 previous similar messages [ 5824.485710] Lustre: server umount lustre-MDT0000 complete [ 5826.204549] LustreError: 232288:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452537 with bad export cookie 14560473634327985374 [ 5826.211868] LustreError: 232288:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5826.211887] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5826.386226] Lustre: server umount lustre-MDT0001 complete [ 5830.529565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.796247] LustreError: 232963:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5830.804638] LustreError: 232963:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 5830.829146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.831992] Lustre: Skipped 2 previous similar messages [ 5832.615271] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5836.256795] LustreError: 232963:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.921991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5838.718661] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5841.114232] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5842.775104] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5845.986578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5848.658207] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5850.940207] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5994.976577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5994.977491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.980517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.982918] Lustre: Skipped 2 previous similar messages [ 5994.986381] Lustre: Skipped 3 previous similar messages [ 6008.287095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6008.306113] LustreError: 235244:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6008.309809] LustreError: 235244:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6008.355505] Lustre: server umount lustre-OST0000 complete [ 6024.159111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6024.227244] Lustre: server umount lustre-MDT0000 complete [ 6025.591091] LustreError: 232990:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452737 with bad export cookie 14560473634327986256 [ 6025.595062] LustreError: 232990:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6025.595257] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6025.731601] Lustre: server umount lustre-MDT0001 complete [ 6115.976735] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6117.098513] Key type lgssc unregistered [ 6117.224457] LNet: 236217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6117.227084] LNetError: 236217:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6117.235496] LNet: Removed LNI 192.168.201.146@tcp [ 6117.553136] Key type .llcrypt unregistered [ 6117.554925] Key type ._llcrypt unregistered [ 6126.060759] Key type ._llcrypt registered [ 6126.061765] Key type .llcrypt registered [ 6126.102310] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6132.620092] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6133.061073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6133.108924] alg: No test for adler32 (adler32-zlib) [ 6133.967174] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 6134.046147] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6135.623172] Key type lgssc registered [ 6135.971126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6151.222525] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6154.965018] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6154.971952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.090694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6156.103418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6156.141296] Lustre: lustre-MDT0000: new disk, initializing [ 6156.168827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.176579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6157.489894] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6161.558697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6161.598848] Lustre: 240204:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6161.620227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6161.621911] Lustre: Skipped 1 previous similar message [ 6161.661268] Lustre: lustre-MDT0001: new disk, initializing [ 6161.692705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6161.704844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6161.708092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6162.984448] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6165.258381] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6167.550325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6167.678172] Lustre: lustre-OST0000: new disk, initializing [ 6167.680671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6167.716456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.640771] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6173.690514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6173.694089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6173.734685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6173.836371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6173.912753] Lustre: lustre-OST0001: new disk, initializing [ 6173.914619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6173.950715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6175.837655] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6180.357307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6180.361276] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6180.397450] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6181.266773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6183.905144] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6192.607860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6192.609941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.613575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.681516] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.682536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.686363] Lustre: Skipped 1 previous similar message [ 6195.689712] Lustre: Skipped 3 previous similar messages [ 6197.960946] LustreError: 243719:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6197.994645] Lustre: server umount lustre-MDT0000 complete [ 6199.185077] LustreError: 243455:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452910 with bad export cookie 8465020531001526442 [ 6199.189114] LustreError: 243455:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6199.189194] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6199.255458] LustreError: 243922:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6199.257619] LustreError: 243922:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6199.303934] Lustre: server umount lustre-MDT0001 complete [ 6210.995091] LustreError: 244122:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6210.997037] LustreError: 244122:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6211.017866] Lustre: server umount lustre-OST0000 complete [ 6222.515559] LustreError: 244324:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6222.517501] LustreError: 244324:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6222.568770] Lustre: server umount lustre-OST0001 complete [ 6227.062689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6230.183214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6232.927564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6236.177524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6242.467219] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6246.021440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.027749] Lustre: lustre-MDT0000: reset Object Index mappings [ 6246.199860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.487108] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6250.352334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6250.360219] Lustre: lustre-MDT0001: reset Object Index mappings [ 6251.764077] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6252.751499] Lustre: 248730:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6255.034276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6255.167651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.169714] Lustre: Skipped 1 previous similar message [ 6256.230125] LustreError: 249085:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.233526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6256.235531] LustreError: 249085:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 6257.102116] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6259.918842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6261.041830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6261.699264] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6264.638431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6265.760986] Lustre: 250574:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6276.064083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6276.067019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.071677] Lustre: Skipped 1 previous similar message [ 6276.073557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6278.016359] LustreError: 250678:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6278.018750] LustreError: 250678:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6278.053927] Lustre: server umount lustre-MDT0000 complete [ 6279.324221] LustreError: 248350:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770452990 with bad export cookie 8465020531001533218 [ 6279.327049] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6279.327799] LustreError: 248350:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6279.452243] Lustre: server umount lustre-MDT0001 complete [ 6290.930941] LustreError: 251081:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6290.932696] LustreError: 251081:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6290.950556] Lustre: server umount lustre-OST0000 complete [ 6302.556303] Lustre: server umount lustre-OST0001 complete [ 6309.027844] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:30:19 (1770453019) [ 6313.792970] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6316.596186] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6327.674120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.765814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6327.777878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6327.816226] Lustre: lustre-MDT0000: new disk, initializing [ 6327.842409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.844673] Lustre: Skipped 1 previous similar message [ 6327.851195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6329.212414] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6333.471961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6333.517547] Lustre: 254790:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6333.628936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6334.962748] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6338.129289] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6339.292805] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6341.639990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6341.743192] Lustre: lustre-OST0000: new disk, initializing [ 6341.744850] Lustre: Skipped 1 previous similar message [ 6341.747319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6341.750432] Lustre: Skipped 2 previous similar messages [ 6343.680301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6343.683061] Lustre: Skipped 1 previous similar message [ 6343.685537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6343.721094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6343.802688] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6346.860901] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6348.182683] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6348.254161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6349.647387] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6349.715853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6353.892184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6353.893429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.895975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.901488] Lustre: Skipped 4 previous similar messages [ 6353.903979] Lustre: Skipped 3 previous similar messages [ 6356.461125] LustreError: 257720:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6356.464139] LustreError: 257720:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6356.506287] Lustre: server umount lustre-OST0000 complete [ 6359.008746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.009221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6359.013440] Lustre: Skipped 1 previous similar message [ 6359.016205] Lustre: Skipped 2 previous similar messages [ 6369.248340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.251162] Lustre: Skipped 3 previous similar messages [ 6372.319125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6372.401893] Lustre: server umount lustre-MDT0000 complete [ 6373.802575] LustreError: 255566:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453085 with bad export cookie 8465020531001535311 [ 6373.806044] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.806859] LustreError: 255566:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6373.946855] Lustre: server umount lustre-MDT0001 complete [ 6380.897411] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6385.115915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.275097] LustreError: 259297:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.294878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.298436] Lustre: Skipped 2 previous similar messages [ 6386.891768] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6390.247336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6391.985159] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6393.113886] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6395.664319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6398.059765] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6401.504854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6401.583954] Lustre: lustre-OST0001: new disk, initializing [ 6401.585997] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6403.336577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6403.341303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6403.389864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6403.802524] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6408.528182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6409.862114] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6416.856426] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6416.865083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6419.939054] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.940182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6419.943382] Lustre: Skipped 3 previous similar messages [ 6419.945128] Lustre: Skipped 2 previous similar messages [ 6424.903183] LustreError: 262797:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6424.905567] LustreError: 262797:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6424.943763] Lustre: server umount lustre-MDT0000 complete [ 6425.056492] LustreError: 260590:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.061052] LustreError: 260590:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 3 previous similar messages [ 6426.288474] LustreError: 259278:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453137 with bad export cookie 8465020531001536193 [ 6426.293053] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6426.294973] LustreError: 259278:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6426.433629] Lustre: server umount lustre-MDT0001 complete [ 6438.408369] Lustre: server umount lustre-OST0000 complete [ 6450.216095] Lustre: server umount lustre-OST0001 complete [ 6456.325018] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6460.176016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.346717] LustreError: 264575:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.368755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.371238] Lustre: Skipped 3 previous similar messages [ 6461.908502] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6465.211833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6466.828865] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6467.941922] Lustre: 265680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6470.620901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6473.017066] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6474.856933] LustreError: 266615:0:(ldlm_lib.c:1185:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.369049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6478.571935] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6485.929090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6485.930141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6492.479378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6493.775765] Lustre: 267552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6500.654578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6500.658670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6504.025947] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:33:34 (1770453214) [ 6516.709805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6516.711121] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.712369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.712378] Lustre: Skipped 1 previous similar message [ 6516.712466] LustreError: Skipped 1 previous similar message [ 6516.717752] Lustre: Skipped 1 previous similar message [ 6517.568380] LustreError: 268257:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6517.571339] LustreError: 268257:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6517.603034] Lustre: server umount lustre-MDT0000 complete [ 6518.840917] LustreError: 264603:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453230 with bad export cookie 8465020531001542563 [ 6518.847097] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.008964] Lustre: server umount lustre-MDT0001 complete [ 6537.594637] Lustre: server umount lustre-OST0000 complete [ 6537.631082] Lustre: 237734:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770453233/real 1770453233] req@ffff96dd075ea680 x1856454362644352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770453249 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6544.989420] Lustre: server umount lustre-OST0001 complete [ 6547.092833] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6550.618946] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6566.534732] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6569.916933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.019279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6570.031505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6570.073478] Lustre: lustre-MDT0000: new disk, initializing [ 6570.104380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6571.416965] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6575.700779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6575.743603] Lustre: 271728:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6575.850830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6577.228660] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6581.759262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6582.357171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6583.687993] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6587.731700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6588.966379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6589.596441] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6594.122667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6604.628260] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 03:35:14 (1770453314) [ 6606.816236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6606.819077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.823171] Lustre: Skipped 1 previous similar message [ 6606.825100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6606.827068] Lustre: Skipped 3 previous similar messages [ 6611.486385] Lustre: server umount lustre-MDT0000 complete [ 6612.803323] LustreError: 272507:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453324 with bad export cookie 8465020531001550515 [ 6612.805282] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6612.806963] LustreError: 272507:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6612.926431] Lustre: server umount lustre-MDT0001 complete [ 6623.693110] Lustre: server umount lustre-OST0000 complete [ 6635.232913] Lustre: server umount lustre-OST0001 complete [ 6641.341744] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6644.617194] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6656.660167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.753321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6656.758085] Lustre: Skipped 2 previous similar messages [ 6656.768985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6656.771862] Lustre: Skipped 4 previous similar messages [ 6656.808682] Lustre: lustre-MDT0000: new disk, initializing [ 6656.810277] Lustre: Skipped 3 previous similar messages [ 6656.835100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.837874] Lustre: Skipped 7 previous similar messages [ 6656.846035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6656.849436] Lustre: Skipped 3 previous similar messages [ 6658.235775] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6662.549366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6662.593964] Lustre: 278766:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6662.699988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6662.704280] Lustre: Skipped 2 previous similar messages [ 6664.058866] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6667.094345] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6668.210344] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6670.477061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6672.156104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6672.427572] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6675.427117] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6676.107730] Lustre: Mounted lustre-client [ 6676.421126] LustreError: 281264:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff96dd096cc000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6676.426135] LustreError: 281264:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6676.427861] LustreError: 281264:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 6676.446845] Lustre: Unmounted lustre-client [ 6681.570755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.577783] Lustre: Skipped 4 previous similar messages [ 6685.518758] Lustre: server umount lustre-OST0000 complete [ 6692.823343] Lustre: server umount lustre-MDT0000 complete [ 6694.032429] LustreError: 278757:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453405 with bad export cookie 8465020531001552048 [ 6694.036316] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.037475] LustreError: 278757:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6694.144966] Lustre: server umount lustre-MDT0001 complete [ 6697.581177] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6698.531073] Key type lgssc unregistered [ 6739.231191] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp [ 6739.615364] LNet: 282541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6739.617872] LNetError: 282541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6805.157301] LNet: Removed LNI 192.168.201.146@tcp [ 6805.460139] Key type .llcrypt unregistered [ 6805.461385] Key type ._llcrypt unregistered [ 6811.386294] Key type ._llcrypt registered [ 6811.387382] Key type .llcrypt registered [ 6811.422319] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6817.672360] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6818.033259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6818.072807] alg: No test for adler32 (adler32-zlib) [ 6818.930547] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 6819.013678] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6820.591119] Key type lgssc registered [ 6820.990431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6831.282429] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:39:01 (1770453541) [ 6833.593466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6833.599453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6834.695961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6834.708576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6834.748488] Lustre: lustre-MDT0000: new disk, initializing [ 6834.775659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.783759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6836.137491] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6840.416023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6840.460114] Lustre: 285697:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6840.484709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6840.487166] Lustre: Skipped 1 previous similar message [ 6840.527690] Lustre: lustre-MDT0001: new disk, initializing [ 6840.563699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6840.576218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6840.579734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6841.972502] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6845.139281] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6846.327912] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6848.573348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6848.685673] Lustre: lustre-OST0000: new disk, initializing [ 6848.688613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6848.720247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6850.718615] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6853.120521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6853.123895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6853.177604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6853.801163] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6858.210467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6858.210919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.212892] LustreError: Skipped 1 previous similar message [ 6858.213367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6858.217284] Lustre: Skipped 1 previous similar message [ 6859.758131] LustreError: 288129:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6859.802306] Lustre: server umount lustre-OST0000 complete [ 6861.280053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6861.283148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.287132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6861.290063] Lustre: Skipped 1 previous similar message [ 6863.165933] LustreError: 288329:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6863.167711] LustreError: 288329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6863.194758] Lustre: server umount lustre-MDT0000 complete [ 6863.328881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.479279] LustreError: 286476:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770453576 with bad export cookie 8302005789271960129 [ 6864.482237] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.483367] LustreError: 286476:0:(ldlm_lockd.c:2597:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6864.555894] LustreError: 288531:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6864.557717] LustreError: 288531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6864.613277] Lustre: server umount lustre-MDT0001 complete [ 6868.380215] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6869.418857] Key type lgssc unregistered [ 6869.555399] LNet: 289061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6869.558093] LNetError: 289061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6869.565126] LNet: Removed LNI 192.168.201.146@tcp [ 6869.864749] Key type .llcrypt unregistered [ 6869.866035] Key type ._llcrypt unregistered [ 6904.605297] Key type ._llcrypt registered [ 6904.606838] Key type .llcrypt registered [ 6904.647197] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6904.751201] Key type .llcrypt unregistered [ 6904.752742] Key type ._llcrypt unregistered [ 6932.291503] Key type ._llcrypt registered [ 6932.292439] Key type .llcrypt registered [ 6932.328431] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6932.432680] Key type .llcrypt unregistered [ 6932.433868] Key type ._llcrypt unregistered [ 6943.299084] Key type ._llcrypt registered [ 6943.300110] Key type .llcrypt registered [ 6943.338953] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6943.450267] Key type .llcrypt unregistered [ 6943.451312] Key type ._llcrypt unregistered [ 6954.766304] Key type ._llcrypt registered [ 6954.767732] Key type .llcrypt registered [ 6954.807972] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6954.913289] Key type .llcrypt unregistered [ 6954.914350] Key type ._llcrypt unregistered [ 6981.394441] Key type ._llcrypt registered [ 6981.395600] Key type .llcrypt registered [ 6981.431668] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6981.527496] Key type .llcrypt unregistered [ 6981.528641] Key type ._llcrypt unregistered [ 6992.591397] Key type ._llcrypt registered [ 6992.592450] Key type .llcrypt registered [ 6992.628538] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6992.735393] Key type .llcrypt unregistered [ 6992.736487] Key type ._llcrypt unregistered [ 7003.917165] Key type ._llcrypt registered [ 7003.918342] Key type .llcrypt registered [ 7003.957573] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7004.054210] Key type .llcrypt unregistered [ 7004.055404] Key type ._llcrypt unregistered [ 7015.027146] Key type ._llcrypt registered [ 7015.028463] Key type .llcrypt registered [ 7015.067264] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7015.169327] Key type .llcrypt unregistered [ 7015.170536] Key type ._llcrypt unregistered [ 7026.122389] Key type ._llcrypt registered [ 7026.123551] Key type .llcrypt registered [ 7026.163105] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7026.269238] Key type .llcrypt unregistered [ 7026.270557] Key type ._llcrypt unregistered [ 7037.151308] Key type ._llcrypt registered [ 7037.152383] Key type .llcrypt registered [ 7037.190445] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7037.285276] Key type .llcrypt unregistered [ 7037.286347] Key type ._llcrypt unregistered [ 7058.072727] Key type ._llcrypt registered [ 7058.073964] Key type .llcrypt registered [ 7058.109633] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7058.208469] Key type .llcrypt unregistered [ 7058.210142] Key type ._llcrypt unregistered [ 7069.019688] Key type ._llcrypt registered [ 7069.020820] Key type .llcrypt registered [ 7069.060723] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7069.161409] Key type .llcrypt unregistered [ 7069.162635] Key type ._llcrypt unregistered [ 7079.991553] Key type ._llcrypt registered [ 7079.992647] Key type .llcrypt registered [ 7080.033392] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7080.143370] Key type .llcrypt unregistered [ 7080.145042] Key type ._llcrypt unregistered [ 7099.201108] Key type ._llcrypt registered [ 7099.202303] Key type .llcrypt registered [ 7099.240417] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 7105.472860] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7105.898768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7105.906640] alg: No test for adler32 (adler32-zlib) [ 7106.756403] Lustre: Lustre: Build Version: 2.17.50_44_g630b48b [ 7106.852515] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7108.431149] Key type lgssc registered [ 7108.807408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7118.694639] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6999 sec ========== 03:43:49 (1770453829) [ 7119.227964] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:43:49 (1770453829) === [ 7120.306925] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:43:50 (1770453830) === [ 7132.818215] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7133.814770] Key type lgssc unregistered [ 7133.942355] LNet: 310743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7133.945298] LNetError: 310743:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7133.953274] LNet: Removed LNI 192.168.201.146@tcp [ 7134.226462] Key type .llcrypt unregistered [ 7134.227676] Key type ._llcrypt unregistered