[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 642473622 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001020] APIC: Switch to symmetric I/O mode setup [ 0.003116] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007774] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.019743] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023654] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025038] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] *** VALIDATE tmpfs *** [ 0.028570] *** VALIDATE proc *** [ 0.029237] *** VALIDATE cgroup *** [ 0.030015] *** VALIDATE cgroup2 *** [ 0.032204] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039265] debug: unmapping init [mem 0xffffffffb6059000-0xffffffffb6060fff] [ 0.041000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041664] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042022] ... version: 2 [ 0.043021] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045009] ... value mask: 0000ffffffffffff [ 0.046025] ... max period: 00007fffffffffff [ 0.047020] ... fixed-purpose events: 3 [ 0.048023] ... event mask: 000000070000000f [ 0.050317] rcu: Hierarchical SRCU implementation. [ 0.054910] smp: Bringing up secondary CPUs ... [ 0.055808] x86: Booting SMP configuration: [ 0.056192] .... node #0, CPUs: #1 #2 #3 [ 0.067393] smp: Brought up 1 node, 4 CPUs [ 0.069014] smpboot: Max logical packages: 1 [ 0.070025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127603] node 0 deferred pages initialised in 55ms [ 0.135013] devtmpfs: initialized [ 0.137276] x86/mm: Memory block size: 128MB [ 0.143007] gcov: version magic: 0x41383552 [ 0.147487] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149875] pinctrl core: initialized pinctrl subsystem [ 0.150229] [ 0.150737] ************************************************************* [ 0.151016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.153029] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154012] ** ** [ 0.155010] ** This means that this kernel is built to expose internal ** [ 0.156011] ** IOMMU data structures, which may compromise security on ** [ 0.157010] ** your system. ** [ 0.158011] ** ** [ 0.159015] ** If you see this message and you are not debugging the ** [ 0.160012] ** kernel, report this immediately to your vendor! ** [ 0.161011] ** ** [ 0.162013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163017] ************************************************************* [ 0.165134] NET: Registered protocol family 16 [ 0.166703] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168133] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170044] cpuidle: using governor menu [ 0.172083] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177576] PCI: Using configuration type 1 for base access [ 0.180137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197371] cryptd: max_cpu_qlen set to 1000 [ 0.201926] ACPI: Added _OSI(Module Device) [ 0.203012] ACPI: Added _OSI(Processor Device) [ 0.205010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212398] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220772] ACPI: Interpreter enabled [ 0.221054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.222009] ACPI: Using IOAPIC for interrupt routing [ 0.223171] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224803] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234763] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.239501] acpiphp: Slot [2] registered [ 0.240147] acpiphp: Slot [3] registered [ 0.241108] acpiphp: Slot [4] registered [ 0.242124] acpiphp: Slot [5] registered [ 0.243221] acpiphp: Slot [6] registered [ 0.244184] acpiphp: Slot [7] registered [ 0.245208] acpiphp: Slot [8] registered [ 0.246215] acpiphp: Slot [9] registered [ 0.247233] acpiphp: Slot [10] registered [ 0.248323] acpiphp: Slot [11] registered [ 0.249083] acpiphp: Slot [12] registered [ 0.250083] acpiphp: Slot [13] registered [ 0.251074] acpiphp: Slot [14] registered [ 0.252096] acpiphp: Slot [15] registered [ 0.253088] acpiphp: Slot [16] registered [ 0.254079] acpiphp: Slot [17] registered [ 0.255097] acpiphp: Slot [18] registered [ 0.256088] acpiphp: Slot [19] registered [ 0.257214] acpiphp: Slot [20] registered [ 0.258256] acpiphp: Slot [21] registered [ 0.259187] acpiphp: Slot [22] registered [ 0.261083] acpiphp: Slot [23] registered [ 0.262092] acpiphp: Slot [24] registered [ 0.263104] acpiphp: Slot [25] registered [ 0.264752] acpiphp: Slot [26] registered [ 0.265102] acpiphp: Slot [27] registered [ 0.266067] acpiphp: Slot [28] registered [ 0.267129] acpiphp: Slot [29] registered [ 0.268067] acpiphp: Slot [30] registered [ 0.269077] acpiphp: Slot [31] registered [ 0.270067] PCI host bridge to bus 0000:00 [ 0.271014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.272025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.273029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.274030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.275020] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.276033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280401] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.283479] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.289015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.293031] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.294014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.295015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.296203] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297602] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.298745] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.300726] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.302968] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.310820] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.312828] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.316064] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.319027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.322017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.329015] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.334760] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.337013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.340025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.347014] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.353105] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.356014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.359015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.366016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.372419] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.375014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.378015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.385013] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.391660] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.394033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.397012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.405030] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.411830] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.414013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.417018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.425012] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.433764] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.434593] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.435557] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.436357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.437194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.440059] iommu: Default domain type: Passthrough [ 0.441532] SCSI subsystem initialized [ 0.445152] ACPI: bus type USB registered [ 0.449152] usbcore: registered new interface driver usbfs [ 0.453083] usbcore: registered new interface driver hub [ 0.455103] usbcore: registered new device driver usb [ 0.457151] pps_core: LinuxPPS API ver. 1 registered [ 0.461010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.466064] PTP clock support registered [ 0.468175] EDAC MC: Ver: 3.0.0 [ 0.470069] PCI: Using ACPI for IRQ routing [ 0.472064] NetLabel: Initializing [ 0.473008] NetLabel: domain hash size = 128 [ 0.474010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.475095] NetLabel: unlabeled traffic allowed by default [ 0.476141] vgaarb: loaded [ 0.477244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.478009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.485000] clocksource: Switched to clocksource kvm-clock [ 0.636182] VFS: Disk quotas dquot_6.6.0 [ 0.640733] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.647560] *** VALIDATE ramfs *** [ 0.652732] *** VALIDATE hugetlbfs *** [ 0.657670] pnp: PnP ACPI init [ 0.662559] pnp: PnP ACPI: found 6 devices [ 0.698289] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.706503] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.708485] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.717557] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.725145] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.728756] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.737808] NET: Registered protocol family 2 [ 0.739805] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.747886] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.757858] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.769406] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.778717] TCP: Hash tables configured (established 65536 bind 65536) [ 0.781644] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.789465] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.792923] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.796910] NET: Registered protocol family 1 [ 0.805037] RPC: Registered named UNIX socket transport module. [ 0.808323] RPC: Registered udp transport module. [ 0.810027] RPC: Registered tcp transport module. [ 0.811401] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.813579] NET: Registered protocol family 44 [ 0.818407] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.821723] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.825602] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.832849] PCI: CLS 0 bytes, default 64 [ 0.835207] Unpacking initramfs... [ 2.854413] debug: unmapping init [mem 0xffff9396fcc54000-0xffff9396fffbffff] [ 2.859735] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.861616] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.863867] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.530452] Initialise system trusted keyrings [ 3.540220] Key type blacklist registered [ 3.541853] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.559033] zbud: loaded [ 3.562990] *** VALIDATE nfs *** [ 3.564130] *** VALIDATE nfs4 *** [ 3.567753] pstore: using deflate compression [ 3.571161] Platform Keyring initialized [ 3.725189] NET: Registered protocol family 38 [ 3.726758] Key type asymmetric registered [ 3.727936] Asymmetric key parser 'x509' registered [ 3.733941] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.749821] io scheduler mq-deadline registered [ 3.751755] io scheduler kyber registered [ 3.753219] io scheduler bfq registered [ 3.756516] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.758898] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.765150] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.768520] ACPI: Power Button [PWRF] [ 3.882345] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.986062] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.208246] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.302260] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.506080] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.546209] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.579260] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.583902] Non-volatile memory driver v1.3 [ 4.585364] Linux agpgart interface v0.103 [ 4.613615] virtio_blk virtio1: [vda] 133544 512-byte logical blocks (68.4 MB/65.2 MiB) [ 4.617082] vda: detected capacity change from 0 to 68374528 [ 4.645237] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.651871] vdb: detected capacity change from 0 to 1073741824 [ 4.686712] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.689370] vdc: detected capacity change from 0 to 2621440000 [ 4.717160] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.723160] vdd: detected capacity change from 0 to 2621440000 [ 4.755513] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.759797] vde: detected capacity change from 0 to 4294967296 [ 4.780160] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.783284] vdf: detected capacity change from 0 to 4294967296 [ 4.791099] libphy: Fixed MDIO Bus: probed [ 4.799577] usbcore: registered new interface driver usbserial_generic [ 4.801734] usbserial: USB Serial support registered for generic [ 4.806618] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.811434] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.813338] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.815605] mousedev: PS/2 mouse device common for all mice [ 4.817931] rtc_cmos 00:05: RTC can wake from S4 [ 4.822606] rtc_cmos 00:05: registered as rtc0 [ 4.824958] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.827349] intel_pstate: CPU model not supported [ 4.832852] hid: raw HID events driver (C) Jiri Kosina [ 4.834873] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.837270] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.839564] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.849054] usbcore: registered new interface driver usbhid [ 4.851110] usbhid: USB HID core driver [ 4.853789] drop_monitor: Initializing network drop monitor service [ 4.856761] Initializing XFRM netlink socket [ 4.858600] NET: Registered protocol family 10 [ 4.864440] Segment Routing with IPv6 [ 4.865651] NET: Registered protocol family 17 [ 4.868742] mpls_gso: MPLS GSO support [ 4.874624] RAS: Correctable Errors collector initialized. [ 4.876702] AVX version of gcm_enc/dec engaged. [ 4.878630] AES CTR mode by8 optimization enabled [ 5.016615] sched_clock: Marking stable (5016340665, 0)->(7034015080, -2017674415) [ 5.031164] registered taskstats version 1 [ 5.033176] Loading compiled-in X.509 certificates [ 5.035568] zswap: loaded using pool lzo/zbud [ 5.067725] Key type big_key registered [ 5.091284] Key type encrypted registered [ 5.093790] ima: No TPM chip found, activating TPM-bypass! [ 5.095906] ima: Allocated hash algorithm: sha1 [ 5.097823] ima: No architecture policies found [ 5.102781] evm: Initialising EVM extended attributes: [ 5.104454] evm: security.selinux [ 5.105441] evm: security.ima [ 5.106707] evm: security.capability [ 5.107786] evm: HMAC attrs: 0x1 [ 5.115141] rtc_cmos 00:05: setting system clock to 2025-08-12 22:56:14 UTC (1755039374) [ 5.123973] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 5.126836] debug: unmapping init [mem 0xffffffffb5d82000-0xffffffffb6058fff] [ 5.134112] Write protecting the kernel read-only data: 28672k [ 5.144147] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 5.146815] debug: unmapping init [mem 0xffffffffb4d14000-0xffffffffb4dfffff] [ 5.198415] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.208251] systemd[1]: Detected virtualization kvm. [ 5.211191] systemd[1]: Detected architecture x86-64. [ 5.216744] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.243444] systemd[1]: No hostname configured. [ 5.244874] systemd[1]: Set hostname to . [ 5.246748] random: systemd: uninitialized urandom read (16 bytes read) [ 5.250437] systemd[1]: Initializing machine ID from random generator. [ 5.570827] random: systemd: uninitialized urandom read (16 bytes read) [ 5.574071] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.583181] random: systemd: uninitialized urandom read (16 bytes read) [ 5.587313] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.597792] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.773096] device-mapper: uevent: version 1.0.3 [ 6.774925] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 8.130660] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.220886] random: fast init done Starting dracut initqueue hook... [ 8.586418] scsi host0: ata_piix [ 8.638154] scsi host1: ata_piix [ 8.644932] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.647059] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.642115] random: crng init done [ 13.647773] random: 7 urandom warning(s) missed due to ratelimiting [ 14.054561] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.812455] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.845168] printk: systemd: 25 output lines suppressed due to ratelimiting [ 18.433088] SELinux: Disabled at runtime. [ 18.532463] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.548512] systemd[1]: Detected virtualization kvm. [ 18.550801] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.625365] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.639735] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.654547] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.665652] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.669461] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.702979] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.751947] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ 20.231947] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. [ 20.443313] hrtimer: interrupt took 2823527 ns Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.842405] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.638039] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.654797] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.925534] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.976575] EDAC sbridge: Ver: 1.1.2 [ 25.748642] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 26.660434] NFS: Registering the id_resolver key type [ 26.662332] Key type id_resolver registered [ 26.664081] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 56.103189] libcfs: loading out-of-tree module taints kernel. [ 56.113910] Key type ._llcrypt registered [ 56.115589] Key type .llcrypt registered [ 56.154435] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 63.005395] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 63.562827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.569091] alg: No test for adler32 (adler32-zlib) [ 64.543122] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 64.835949] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 64.837176] LNet: Accept secure, port 988 [ 66.431108] Key type lgssc registered [ 66.919734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.105983] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.026689] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.354350] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.773241] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.392437] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.689324] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 91.957211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.977905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.988839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.079626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 93.094703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.143636] Lustre: lustre-MDT0000: new disk, initializing [ 93.171971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.179785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 94.517204] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.632152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.662292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.688515] Lustre: 6447:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.697431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.700062] Lustre: Skipped 1 previous similar message [ 99.735485] Lustre: lustre-MDT0001: new disk, initializing [ 99.754793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.765365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.768719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 101.035273] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.267292] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 106.350200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.375506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.452882] Lustre: lustre-OST0000: new disk, initializing [ 106.454351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 106.471277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 107.626212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 107.629678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.650453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 108.224198] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.678835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.699756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.737229] Lustre: lustre-OST0001: new disk, initializing [ 113.739531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 113.756043] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 115.709783] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.707660] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 122.920351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 122.923633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 122.947368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 125.249599] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 132.035296] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 133.574457] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 135.345270] Lustre: DEBUG MARKER: Client: 2.16.56.30 [ 136.359422] Lustre: DEBUG MARKER: MDS: 2.16.56.30 [ 137.299945] Lustre: DEBUG MARKER: OSS: 2.16.56.30 [ 137.876935] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 12 18:58:26 EDT 2025 [ 143.869023] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 144.374949] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 156.127529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 156.129688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.134625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.687392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.687594] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 158.689347] Lustre: Skipped 1 previous similar message [ 158.693331] Lustre: Skipped 1 previous similar message [ 162.043958] LustreError: 12104:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 162.078514] Lustre: server umount lustre-MDT0000 complete [ 163.321833] LustreError: 8341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039532 with bad export cookie 8767475794756991298 [ 163.323461] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 163.326461] LustreError: 8341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 163.807627] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.807868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 163.813213] Lustre: Skipped 2 previous similar messages [ 163.815162] Lustre: Skipped 2 previous similar messages [ 168.927439] LustreError: 8349:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 168.927547] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.933769] LustreError: 8349:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 168.935678] Lustre: Skipped 2 previous similar messages [ 169.462553] LustreError: 12306:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 169.464586] LustreError: 12306:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 169.528995] Lustre: server umount lustre-MDT0001 complete [ 170.773107] LustreError: 12507:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 170.775481] LustreError: 12507:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 170.793547] Lustre: server umount lustre-OST0000 complete [ 172.063873] Lustre: server umount lustre-OST0001 complete [ 174.049029] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 176.921477] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 180.199400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.439746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.425902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.607740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.116662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.140407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.227675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.239754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.274341] Lustre: lustre-MDT0000: new disk, initializing [ 189.299967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.307730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.790180] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 195.133710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.155744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.195341] Lustre: 14870:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.314232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.694366] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 199.899488] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 201.111500] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 203.271324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.291670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 203.387218] Lustre: lustre-OST0000: new disk, initializing [ 203.388312] Lustre: Skipped 1 previous similar message [ 203.389879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.391976] Lustre: Skipped 2 previous similar messages [ 203.418413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.421206] Lustre: Skipped 1 previous similar message [ 205.192606] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 205.304552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.307409] Lustre: Skipped 1 previous similar message [ 205.309239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.335304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.305923] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 209.717177] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 209.808344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 211.107196] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 211.181715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 213.870116] LustreError: 17773:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 213.872374] LustreError: 17773:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 213.939877] Lustre: server umount lustre-OST0000 complete [ 215.520948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.521722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 215.524930] Lustre: Skipped 1 previous similar message [ 215.528748] Lustre: Skipped 1 previous similar message [ 224.738757] LustreError: 17974:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 224.740634] LustreError: 17974:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 224.765078] Lustre: server umount lustre-MDT0000 complete [ 225.760591] LustreError: 14875:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 225.766992] LustreError: 14875:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 226.012381] LustreError: 14861:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039595 with bad export cookie 8767475794756992719 [ 226.015465] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 226.017779] LustreError: 14861:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 236.514978] LustreError: 18176:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 236.516538] LustreError: 18176:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 236.587028] Lustre: server umount lustre-MDT0001 complete [ 238.383976] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:00:07 (1755039607) [ 244.557837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.701179] LustreError: 19311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 244.722923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.973793] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 248.697509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 250.008627] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 251.731640] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 252.941748] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 253.658234] Lustre: Failing over lustre-MDT0000 [ 253.661605] LustreError: 19340:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 253.663593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.664564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 253.664569] Lustre: Skipped 2 previous similar messages [ 253.666486] LustreError: 19340:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 253.670493] Lustre: Skipped 1 previous similar message [ 253.678069] LustreError: 19340:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 253.739423] LustreError: 20641:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 253.742079] LustreError: 20641:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 253.767597] Lustre: server umount lustre-MDT0000 complete [ 258.784284] LustreError: 19311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.788211] LustreError: 19311:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 264.160091] LustreError: 19311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.166424] LustreError: 19311:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 266.836802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.891427] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.389766] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 271.651502] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 272.352459] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 272.357760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 272.364538] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 273.679153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.679728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.683668] Lustre: Skipped 1 previous similar message [ 273.687248] Lustre: Skipped 2 previous similar messages [ 279.825032] Lustre: server umount lustre-MDT0000 complete [ 280.033152] LustreError: 20016:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.039965] LustreError: 20016:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 281.162179] LustreError: 21616:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039650 with bad export cookie 8767475794756994126 [ 281.165628] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 281.165985] LustreError: 21616:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 281.308350] Lustre: server umount lustre-MDT0001 complete [ 284.412480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.621927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 284.625779] Lustre: Skipped 2 previous similar messages [ 286.079608] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 289.159984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 290.735107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 292.593630] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 293.857740] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 294.647177] Lustre: Failing over lustre-MDT0000 [ 294.719179] LustreError: 23635:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 294.720905] LustreError: 23635:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 294.748356] Lustre: server umount lustre-MDT0000 complete [ 298.784702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.785440] LustreError: 23600:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.790275] Lustre: Skipped 1 previous similar message [ 298.797409] LustreError: 23600:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 307.964789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 308.004693] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.444777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 312.766499] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 313.312248] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 313.316016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 313.318202] Lustre: Skipped 1 previous similar message [ 313.323709] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 318.431737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.435844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.438285] Lustre: Skipped 2 previous similar messages [ 329.183159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 329.233944] Lustre: server umount lustre-MDT0000 complete [ 330.444390] LustreError: 23004:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039699 with bad export cookie 8767475794756995057 [ 330.448533] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.449470] LustreError: 23004:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 330.589274] Lustre: server umount lustre-MDT0001 complete [ 333.442442] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:01:42 (1755039702) [ 338.620384] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 341.536972] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 344.920672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.268168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.352479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.530802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.701219] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 358.342354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.364041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.451325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 358.463899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 358.500149] Lustre: lustre-MDT0000: new disk, initializing [ 358.525984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.528208] Lustre: Skipped 2 previous similar messages [ 358.534293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 359.859385] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 364.010296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.035016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 364.066117] Lustre: 28251:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 364.165174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 365.417256] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 368.458747] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 369.645972] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 371.859301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 371.886855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 373.774807] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 373.806772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 373.814350] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 374.026733] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 377.166308] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 379.247869] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 383.970921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 383.971123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.973523] LustreError: Skipped 1 previous similar message [ 383.979158] Lustre: Skipped 2 previous similar messages [ 383.980980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 383.983715] Lustre: Skipped 5 previous similar messages [ 385.466091] LustreError: 30867:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 385.468065] LustreError: 30867:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 385.533506] Lustre: server umount lustre-OST0000 complete [ 401.375097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 401.440878] Lustre: server umount lustre-MDT0000 complete [ 402.787443] LustreError: 29983:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039772 with bad export cookie 8767475794756995456 [ 402.791952] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 402.792354] LustreError: 29983:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 402.994598] Lustre: server umount lustre-MDT0001 complete [ 408.065913] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 409.313105] Key type lgssc unregistered [ 409.483277] LNet: 31796:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.533380] LNet: Removed LNI 192.168.202.116@tcp [ 410.912345] Key type .llcrypt unregistered [ 410.914078] Key type ._llcrypt unregistered [ 420.488794] Key type ._llcrypt registered [ 420.490015] Key type .llcrypt registered [ 420.524842] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 420.818511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 420.839889] alg: No test for adler32 (adler32-zlib) [ 421.691378] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 421.774640] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 421.776407] LNet: Accept secure, port 988 [ 423.359165] Key type lgssc registered [ 423.708653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 426.614913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 426.623621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.796307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.113693] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 432.008868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 432.133886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.514140] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 435.452837] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 436.750510] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 439.130436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 439.260228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.270032] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 442.345884] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 443.064830] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 459.286251] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:03:48 (1755039828) [ 460.005344] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 460.667453] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 462.952192] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:03:51 (1755039831) [ 465.885814] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:03:54 (1755039834) [ 466.263833] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 469.602782] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 469.605734] Lustre: Skipped 2 previous similar messages [ 470.274412] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:03:59 (1755039839) [ 473.985369] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 473.989558] Lustre: Skipped 7 previous similar messages [ 478.864876] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:04:07 (1755039847) [ 479.861114] Lustre: *** cfs_fail_loc=131b, val=2*** [ 481.097124] Lustre: *** cfs_fail_loc=131b, val=2*** [ 483.853274] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:04:12 (1755039852) [ 484.474266] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 484.476967] Lustre: Skipped 6 previous similar messages [ 488.617495] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:04:17 (1755039857) [ 493.538664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.542114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.547512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.561402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.562843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.562889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.570289] Lustre: Skipped 1 previous similar message [ 498.283101] LustreError: 39697:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 498.366506] Lustre: server umount lustre-OST0000 complete [ 499.684669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.688226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 499.689081] Lustre: Skipped 1 previous similar message [ 504.800311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.802815] Lustre: Skipped 2 previous similar messages [ 505.788915] LustreError: 39900:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 505.790735] LustreError: 39900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 505.813102] Lustre: server umount lustre-MDT0000 complete [ 507.028194] LustreError: 33136:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039876 with bad export cookie 6568309490835383130 [ 507.031569] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.032586] LustreError: 33136:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.096995] LustreError: 40101:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 507.098705] LustreError: 40101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 507.179085] Lustre: server umount lustre-MDT0001 complete [ 511.757388] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 512.789876] Key type lgssc unregistered [ 512.919373] LNet: 40627:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.957384] LNet: Removed LNI 192.168.202.116@tcp [ 514.236482] Key type .llcrypt unregistered [ 514.238134] Key type ._llcrypt unregistered [ 519.999856] Key type ._llcrypt registered [ 520.001261] Key type .llcrypt registered [ 520.039060] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 526.067554] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 526.549744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 526.568277] alg: No test for adler32 (adler32-zlib) [ 527.428030] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 527.515210] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 527.517034] LNet: Accept secure, port 988 [ 529.095198] Key type lgssc registered [ 529.472660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 532.427976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 534.618550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 536.459666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 538.522843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 540.861413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 540.884600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 540.892288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.981258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 541.993307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 542.028743] Lustre: lustre-MDT0000: new disk, initializing [ 542.052790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.060387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 543.417593] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 547.719661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 547.744523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.781048] Lustre: 43710:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 547.806096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 547.808140] Lustre: Skipped 1 previous similar message [ 547.846127] Lustre: lustre-MDT0001: new disk, initializing [ 547.879951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 547.891193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 547.894205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 549.187193] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 552.299782] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 553.463351] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 555.686824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 555.709565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 555.821026] Lustre: lustre-OST0000: new disk, initializing [ 555.823135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 555.851015] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.123574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 556.126451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 556.163240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 557.697923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 560.676597] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 561.975864] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 562.057448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 563.371120] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 563.448972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 566.126072] LustreError: 46616:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 566.190813] Lustre: server umount lustre-OST0000 complete [ 568.287860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 568.290650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.295890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.872631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.872982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.992243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.994122] Lustre: Skipped 2 previous similar messages [ 576.996749] LustreError: 46817:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 576.999538] LustreError: 46817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 577.025527] Lustre: server umount lustre-MDT0000 complete [ 578.246275] LustreError: 43703:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755039947 with bad export cookie 3029038505139949058 [ 578.249279] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.250114] LustreError: 43703:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.747121] LustreError: 47019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 587.749212] LustreError: 47019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 587.839677] Lustre: server umount lustre-MDT0001 complete [ 590.889983] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:05:59 (1755039959) [ 593.112515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.273602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.481454] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 597.217220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 597.340747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 598.597023] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 600.448687] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 601.663211] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 603.925280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 605.945125] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 607.729166] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.333347] LustreError: 49938:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 625.632698] Lustre: Modifying parameter general.timeout=1 in log params [ 626.809747] Lustre: Modifying parameter general.timeout=5 in log params [ 626.811565] Lustre: Skipped 3 previous similar messages [ 629.084609] Lustre: Modifying parameter general.timeout=13 in log params [ 629.086007] Lustre: Skipped 7 previous similar messages [ 633.334200] Lustre: Modifying parameter general.timeout=28 in log params [ 633.335904] Lustre: Skipped 14 previous similar messages [ 641.420203] Lustre: Modifying parameter general.timeout=55 in log params [ 641.422872] Lustre: Skipped 26 previous similar messages [ 657.618369] Lustre: Modifying parameter general.timeout=112 in log params [ 657.619823] Lustre: Skipped 56 previous similar messages [ 665.596619] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:07:14 (1755040034) [ 666.483963] Lustre: 56834:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 667.692377] Lustre: 57030:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 667.695485] Lustre: 57030:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 677.896354] Lustre: 58665:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 677.900379] Lustre: 58665:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 3 previous similar messages [ 677.903762] LustreError: 58665:0:(mgs_handler.c:990:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 679.181537] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:07:28 (1755040048) [ 684.001772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.004526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.005257] Lustre: Skipped 2 previous similar messages [ 684.006737] Lustre: Skipped 1 previous similar message [ 688.762593] LustreError: 59362:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 688.764365] LustreError: 59362:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 688.792892] Lustre: server umount lustre-MDT0000 complete [ 689.121293] LustreError: 48443:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.127458] LustreError: 48443:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 690.037188] LustreError: 49987:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040059 with bad export cookie 3029038505139949926 [ 690.041799] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.191613] Lustre: server umount lustre-MDT0001 complete [ 700.720137] LustreError: 59765:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 700.722479] LustreError: 59765:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 700.792541] Lustre: server umount lustre-OST0000 complete [ 703.597810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 704.883970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 706.071934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 707.322583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 709.477312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.570276] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 709.579526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 709.582411] Lustre: Skipped 40 previous similar messages [ 709.662718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.664563] Lustre: Skipped 1 previous similar message [ 710.912811] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 713.652201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 713.680913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 713.690147] Lustre: 61212:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 715.068315] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 716.754338] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 717.892059] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 720.207376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 720.282564] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 722.234208] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 723.928948] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 729.905906] Lustre: 63425:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 734.688984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 734.690617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.691666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 734.699583] Lustre: Skipped 1 previous similar message [ 737.962095] LustreError: 63684:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 737.963814] LustreError: 63684:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 738.027838] Lustre: server umount lustre-OST0000 complete [ 739.295753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 739.297591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.928073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 744.930777] Lustre: Skipped 4 previous similar messages [ 745.364645] Lustre: server umount lustre-MDT0000 complete [ 746.538976] LustreError: 61205:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040115 with bad export cookie 3029038505139952215 [ 746.541962] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 746.542702] LustreError: 61205:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 746.676569] Lustre: server umount lustre-MDT0001 complete [ 751.475710] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 752.529546] Key type lgssc unregistered [ 752.670543] LNet: 64612:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.701406] LNet: Removed LNI 192.168.202.116@tcp [ 754.045440] Key type .llcrypt unregistered [ 754.046416] Key type ._llcrypt unregistered [ 762.969788] Key type ._llcrypt registered [ 762.970803] Key type .llcrypt registered [ 763.013155] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 763.372893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 763.384339] alg: No test for adler32 (adler32-zlib) [ 764.235703] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 764.327378] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 764.328967] LNet: Accept secure, port 988 [ 765.911148] Key type lgssc registered [ 766.265618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 768.804482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.812865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.976563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.190175] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 773.918356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 774.136405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.527739] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 777.341437] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 778.468949] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 780.722092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 780.912579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.880211] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 784.598590] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 802.937877] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 805.858574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.860516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 805.862486] Lustre: Skipped 2 previous similar messages [ 805.864315] Lustre: Skipped 1 previous similar message [ 810.977211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.979267] Lustre: Skipped 2 previous similar messages [ 811.580420] LustreError: 68690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 811.605682] Lustre: server umount lustre-MDT0000 complete [ 812.865079] LustreError: 67593:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040182 with bad export cookie 8357933260812443283 [ 812.868718] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.869115] LustreError: 67593:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 812.945669] LustreError: 68893:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 812.948586] LustreError: 68893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 813.042587] Lustre: server umount lustre-MDT0001 complete [ 824.624905] LustreError: 69093:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 824.626620] LustreError: 69093:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 824.686440] Lustre: server umount lustre-OST0000 complete [ 827.391367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 828.600873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 829.836786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 831.139912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 833.488443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.587156] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 833.596170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 833.691667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.967642] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 837.875816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 837.905632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 837.916543] Lustre: 70541:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 838.093546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.446428] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 841.263029] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 842.461205] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 844.773079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 844.834145] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 846.768615] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 848.506565] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 849.026342] Lustre: 72766:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 849.645554] Lustre: Modifying parameter general.jobid_var=disable in log params [ 852.961695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.963515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.964554] LustreError: Skipped 1 previous similar message [ 852.969558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.973158] Lustre: Skipped 1 previous similar message [ 857.003115] LustreError: 73013:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 857.005869] LustreError: 73013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 857.079443] Lustre: server umount lustre-OST0000 complete [ 858.591947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 858.593950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.596936] Lustre: Skipped 1 previous similar message [ 858.598943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.200839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.201300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.442109] LustreError: 73220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 864.443952] LustreError: 73220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 864.471216] Lustre: server umount lustre-MDT0000 complete [ 865.686410] LustreError: 70532:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040235 with bad export cookie 8357933260812444095 [ 865.689666] LustreError: 70532:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 865.690482] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.839226] Lustre: server umount lustre-MDT0001 complete [ 870.241521] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 871.331761] Key type lgssc unregistered [ 871.487521] LNet: 73947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.549427] LNet: Removed LNI 192.168.202.116@tcp [ 872.884599] Key type .llcrypt unregistered [ 872.885514] Key type ._llcrypt unregistered [ 881.745522] Key type ._llcrypt registered [ 881.746538] Key type .llcrypt registered [ 881.783449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 882.080666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.123091] alg: No test for adler32 (adler32-zlib) [ 882.984841] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 883.070510] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 883.072549] LNet: Accept secure, port 988 [ 884.647190] Key type lgssc registered [ 885.021619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.565475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.572046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.740493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.987603] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 892.825348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 893.013198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.388826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 896.257650] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.521260] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 899.858864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 900.050157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.936254] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 903.730862] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.582523] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 922.154932] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 922.157468] Lustre: Skipped 1 previous similar message [ 923.343569] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 923.345628] Lustre: Skipped 3 previous similar messages [ 925.430934] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 925.432802] Lustre: Skipped 6 previous similar messages [ 929.657919] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 929.660735] Lustre: Skipped 13 previous similar messages [ 937.951061] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 937.952881] Lustre: Skipped 26 previous similar messages [ 954.384758] Lustre: 83039:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 954.682798] Lustre: Disabling parameter general.jobid_var= in log params [ 954.684316] Lustre: Skipped 45 previous similar messages [ 955.205923] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:12:04 (1755040324) [ 955.714434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 956.312240] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:12:05 (1755040325) [ 959.456890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 959.459957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.464587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 960.483592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 960.484895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.486072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.245174] LustreError: 83332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 963.317219] Lustre: server umount lustre-OST0000 complete [ 965.600820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.605096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.608096] Lustre: Skipped 2 previous similar messages [ 970.720872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.723644] Lustre: Skipped 1 previous similar message [ 970.754553] LustreError: 83535:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 970.756379] LustreError: 83535:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 970.790634] Lustre: server umount lustre-MDT0000 complete [ 972.086407] LustreError: 76027:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040341 with bad export cookie 15777478044467685786 [ 972.089569] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.090120] LustreError: 76027:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 972.157964] LustreError: 83738:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 972.160891] LustreError: 83738:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 972.245019] Lustre: server umount lustre-MDT0001 complete [ 977.133159] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 978.367576] Key type lgssc unregistered [ 978.512320] LNet: 84264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.557472] LNet: Removed LNI 192.168.202.116@tcp [ 979.917839] Key type .llcrypt unregistered [ 979.919548] Key type ._llcrypt unregistered [ 981.564551] Key type ._llcrypt registered [ 981.566484] Key type .llcrypt registered [ 981.608080] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 983.115278] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules [ 983.537711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.613496] alg: No test for adler32 (adler32-zlib) [ 984.477703] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 984.567870] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 984.570764] LNet: Accept secure, port 988 [ 986.159169] Key type lgssc registered [ 986.571039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.008899] LustreError: 85364:0:(lu_object.c:1383:lu_context_key_register()) cfs_race id 60d sleeping [ 994.367677] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 997.501345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.223320] LustreError: 85364:0:(lu_object.c:1383:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1008.231706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.419203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.782528] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1013.690431] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:13:02 (1755040382) [ 1015.290561] LustreError: 86395:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1015.294099] LustreError: 86395:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1015.296825] LustreError: 86395:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1015.363575] LustreError: 87163:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1015.460192] Lustre: server umount lustre-MDT0000 complete [ 1020.145597] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1021.265861] Key type lgssc unregistered [ 1021.405381] LNet: 87843:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.437398] LNet: Removed LNI 192.168.202.116@tcp [ 1022.765971] Key type .llcrypt unregistered [ 1022.767596] Key type ._llcrypt unregistered [ 1029.922163] Key type ._llcrypt registered [ 1029.923375] Key type .llcrypt registered [ 1029.958867] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1030.276869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.284264] alg: No test for adler32 (adler32-zlib) [ 1031.138690] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 1031.226977] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1031.229716] LNet: Accept secure, port 988 [ 1032.823171] Key type lgssc registered [ 1033.162096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1035.738946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1035.745452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.922764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.232046] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1041.165888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1041.404522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1042.663774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1044.452897] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1045.764371] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1048.063302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1048.309524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.359538] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1052.314519] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1084.718052] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:14:13 (1755040453) [ 1085.272487] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1085.846333] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:14:14 (1755040454) [ 1087.456344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1087.459153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.464679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1088.480228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1088.481469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.483217] Lustre: Skipped 1 previous similar message [ 1088.486882] Lustre: Skipped 1 previous similar message [ 1093.054710] LustreError: 91915:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1093.080361] Lustre: server umount lustre-MDT0000 complete [ 1093.602262] LustreError: 89727:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.606605] LustreError: 89727:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1094.397579] LustreError: 88990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040463 with bad export cookie 2637271185914906863 [ 1094.401335] LustreError: 88990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1094.401416] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.478837] LustreError: 92118:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1094.480949] LustreError: 92118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1094.575129] Lustre: server umount lustre-MDT0001 complete [ 1106.160135] LustreError: 92319:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1106.162061] LustreError: 92319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1106.315387] Lustre: server umount lustre-OST0000 complete [ 1110.245817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.437065] LustreError: 92948:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.501982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.922491] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1114.933569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1115.191742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1116.673166] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1118.513470] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1119.730390] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1121.784867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1124.007039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1124.031791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1124.106607] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1124.109152] LustreError: 92936:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1124.119301] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1124.122069] LustreError: 94795:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -98 [ 1124.124134] LustreError: 94795:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 1124.125772] LustreError: 94795:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1124.181740] Lustre: server umount lustre-OST0000 complete [ 1124.183170] LustreError: 94795:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1127.077114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1127.154840] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1127.157617] LustreError: 92936:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1127.160531] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1127.163163] LustreError: 95257:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -98 [ 1127.165211] LustreError: 95257:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 1127.167245] LustreError: 95257:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1127.227898] Lustre: server umount lustre-OST0000 complete [ 1127.229959] LustreError: 95257:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1128.220611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1130.637147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1130.718035] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1130.720442] Lustre: Found index 0 for lustre-OST0000, updating log [ 1130.722761] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1130.757882] Lustre: lustre-OST0000: new disk, initializing [ 1130.759672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1130.864091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.941552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1134.256322] Lustre: Failing over lustre-OST0000 [ 1134.272122] LustreError: 96328:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1134.275021] LustreError: 96328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1134.350491] Lustre: server umount lustre-OST0000 complete [ 1135.659051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.663461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.665165] Lustre: Skipped 1 previous similar message [ 1139.745483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.745827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.752025] Lustre: Skipped 1 previous similar message [ 1141.755847] LustreError: 96530:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1141.757855] LustreError: 96530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1141.786530] Lustre: server umount lustre-MDT0000 complete [ 1143.117434] LustreError: 92928:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040512 with bad export cookie 2637271185914909593 [ 1143.120406] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.123918] LustreError: 92928:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.274709] Lustre: server umount lustre-MDT0001 complete [ 1146.198073] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:15:15 (1755040515) [ 1151.953687] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1155.768302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.929241] LustreError: 98089:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.960043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.425489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1160.491289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1162.203727] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1163.242265] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1165.572519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1167.869099] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1170.922520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1170.957693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1170.984706] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1171.028663] Lustre: lustre-OST0001: new disk, initializing [ 1171.030558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1173.159241] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1174.984592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1174.989290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1175.042247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1176.475391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1180.262371] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1189.318641] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 1190.371294] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.374450] Lustre: Skipped 2 previous similar messages [ 1192.833704] LustreError: 101309:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1192.836743] LustreError: 101309:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1192.865735] Lustre: server umount lustre-MDT0001 complete [ 1193.828856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1194.432594] LustreError: 98084:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.220331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1196.259161] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1196.262091] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1196.281344] Lustre: 98077:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1196.550132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1196.552237] Lustre: Skipped 3 previous similar messages [ 1197.798498] LustreError: 102044:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1197.800820] LustreError: 102044:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1197.803548] LustreError: 98081:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1197.835164] LustreError: 102058:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -17 [ 1197.837724] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1197.837724] [ 1197.902948] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1201.150519] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 19:16:10 (1755040570) [ 1201.638892] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 1207.001467] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1213.050306] Lustre: 103897:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1221.157605] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1222.444889] Lustre: 105029:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1273.314094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.314250] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.314633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.314640] Lustre: Skipped 3 previous similar messages [ 1273.323447] Lustre: Skipped 3 previous similar messages [ 1278.935370] LustreError: 105460:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1278.937729] LustreError: 105460:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1278.974413] Lustre: server umount lustre-MDT0000 complete [ 1280.203862] LustreError: 98070:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040649 with bad export cookie 2637271185914910461 [ 1280.205038] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.208650] LustreError: 98070:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1280.376835] Lustre: server umount lustre-MDT0001 complete [ 1290.759223] Lustre: server umount lustre-OST0000 complete [ 1302.279158] Lustre: server umount lustre-OST0001 complete [ 1304.131105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1307.030836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1307.386828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1310.248482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1313.201145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.561959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1319.777339] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1323.333197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.338516] Lustre: lustre-MDT0000: reset Object Index mappings [ 1323.493390] LustreError: 108928:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.497406] LustreError: 108928:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1323.527121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.776370] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1327.700325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1327.706894] Lustre: lustre-MDT0001: reset Object Index mappings [ 1329.371659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1330.386189] Lustre: 110032:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1332.635454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1333.923056] LustreError: 110386:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.926550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1334.839881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1337.673073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1338.926847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1338.930622] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1338.930840] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1339.869400] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1343.145762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1344.395079] Lustre: 111874:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1364.653607] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:18:53 (1755040733) [ 1394.655885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1394.658101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.661825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.663488] Lustre: Skipped 7 previous similar messages [ 1400.013334] LustreError: 112221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1400.015439] LustreError: 112221:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1400.054904] Lustre: server umount lustre-MDT0000 complete [ 1400.290221] LustreError: 108927:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.295324] LustreError: 108927:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1401.342272] LustreError: 112122:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755040770 with bad export cookie 2637271185914957228 [ 1401.346814] LustreError: 112122:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1401.347192] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1407.639363] Lustre: server umount lustre-MDT0001 complete [ 1423.327189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1423.381772] Lustre: server umount lustre-OST0000 complete [ 1430.850963] Lustre: server umount lustre-OST0001 complete [ 1433.007498] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1436.133729] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1440.223876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1442.770621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1444.974942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1447.323107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1449.952846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1449.976150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.075565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1450.087672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1450.132116] Lustre: lustre-MDT0000: new disk, initializing [ 1450.163824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1451.479479] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1454.977670] Lustre: server umount lustre-MDT0000 complete [ 1456.006431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1458.801912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1458.836955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.975584] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1458.982197] Lustre: 116070:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1458.989801] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1459.075760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.078368] Lustre: Skipped 4 previous similar messages [ 1460.397653] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1462.311756] Lustre: server umount lustre-MDT0000 complete [ 1465.525339] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:20:34 (1755040834) [ 1466.089216] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1466.649865] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:20:35 (1755040835) [ 1473.056607] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1476.894912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.464238] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1481.307154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1481.331272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1481.365903] Lustre: 118072:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1481.391422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1481.393607] Lustre: Skipped 1 previous similar message [ 1481.435775] Lustre: lustre-MDT0001: new disk, initializing [ 1481.485283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1481.488548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1482.767936] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1485.052519] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1487.615573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1487.648482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1487.802697] Lustre: lustre-OST0000: new disk, initializing [ 1487.806577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1489.334705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1489.341463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1489.353546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1490.355632] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1495.305782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1495.347250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1497.379555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1497.383143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1497.432488] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1497.838554] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1502.835296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1504.004241] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1737.866505] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:25:06 (1755041106) [ 1753.056381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1753.060079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.065128] Lustre: Skipped 5 previous similar messages [ 1753.067729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.069611] Lustre: Skipped 8 previous similar messages [ 1755.675473] LustreError: 127026:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1755.680064] LustreError: 127026:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1755.757677] Lustre: server umount lustre-MDT0000 complete [ 1757.477908] LustreError: 118064:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755041126 with bad export cookie 2637271185915010568 [ 1757.483627] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1757.484556] LustreError: 118064:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1758.688995] LustreError: 118080:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1758.696037] LustreError: 118080:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 1763.796936] Lustre: server umount lustre-MDT0001 complete [ 1771.198372] Lustre: server umount lustre-OST0000 complete [ 1778.494716] Lustre: server umount lustre-OST0001 complete [ 1780.516674] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1783.902836] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1787.506824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1789.789068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.069630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.186668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1796.461516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1796.484559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.568546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1796.579900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1796.581478] Lustre: Skipped 1 previous similar message [ 1796.620284] Lustre: lustre-MDT0000: new disk, initializing [ 1796.621626] Lustre: Skipped 1 previous similar message [ 1796.645022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.647040] Lustre: Skipped 4 previous similar messages [ 1796.652312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1797.893109] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1802.037208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1802.063163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1802.099675] Lustre: 129795:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1802.223580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1803.711079] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1806.802499] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1808.017986] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1810.210939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1810.234858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1811.706483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1811.742155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1812.329915] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1815.420984] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1818.638984] Lustre: lustre-MDD0000: changelog on [ 1819.576387] Lustre: lustre-MDD0001: changelog on [ 2209.078938] Lustre: lustre-MDD0001: changelog off [ 2209.717188] Lustre: 129801:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2210.062847] Lustre: lustre-MDD0000: changelog off [ 2211.808656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2211.810789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.813853] Lustre: Skipped 5 previous similar messages [ 2211.816761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2211.818406] Lustre: Skipped 7 previous similar messages [ 2213.742158] LustreError: 138997:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2213.745297] LustreError: 138997:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2213.834198] Lustre: server umount lustre-OST0000 complete [ 2221.312504] Lustre: server umount lustre-MDT0000 complete [ 2221.537636] LustreError: 129805:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.544249] LustreError: 129805:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2222.678747] LustreError: 129786:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755041592 with bad export cookie 2637271185915822393 [ 2222.681182] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.682289] LustreError: 129786:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2233.417345] Lustre: server umount lustre-MDT0001 complete [ 2238.461684] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2239.589741] Key type lgssc unregistered [ 2239.736447] LNet: 139928:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.805393] LNet: Removed LNI 192.168.202.116@tcp [ 2241.167651] Key type .llcrypt unregistered [ 2241.168817] Key type ._llcrypt unregistered [ 2247.889599] Key type ._llcrypt registered [ 2247.891113] Key type .llcrypt registered [ 2247.935546] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2257.623366] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2258.562221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.682581] alg: No test for adler32 (adler32-zlib) [ 2259.706021] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 2259.865801] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2259.869306] LNet: Accept secure, port 988 [ 2261.503774] Key type lgssc registered [ 2262.402241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2269.149621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2273.957076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2278.180440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2282.635255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2287.237198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2287.279754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.300867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.501268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2288.529812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2288.598728] Lustre: lustre-MDT0000: new disk, initializing [ 2288.670022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.683785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2291.240810] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2303.357762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2303.505720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2303.618746] Lustre: 143064:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2303.714860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2303.719885] Lustre: Skipped 1 previous similar message [ 2304.061396] Lustre: lustre-MDT0001: new disk, initializing [ 2304.305449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.418267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2304.423062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2313.466102] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2327.944224] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2335.461925] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2346.478577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2346.607738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.101885] Lustre: lustre-OST0000: new disk, initializing [ 2347.117183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2347.257734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.937508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2356.973957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2357.208271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2358.107604] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2368.316747] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2374.817625] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module obdecho/obdecho [ 2376.935139] LustreError: 145665:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2376.941604] LustreError: 145665:0:(obd_config.c:779:class_setup()) setup ec failed (-22) [ 2378.902048] LustreError: 145761:0:(echo_client.c:1878:echo_md_handler()) Can not resolve the path : rc = -2 [ 2391.524852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2391.529348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.543048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.059862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.062276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.014632] LustreError: 146102:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2396.152972] Lustre: server umount lustre-MDT0000 complete [ 2399.564097] LustreError: 144802:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755041768 with bad export cookie 18367158321801095482 [ 2399.564531] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.571899] LustreError: 144802:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2399.724110] LustreError: 146303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2399.726835] LustreError: 146303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2400.021900] Lustre: server umount lustre-MDT0001 complete [ 2412.684088] LustreError: 146506:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2412.719198] LustreError: 146506:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2412.936956] Lustre: server umount lustre-OST0000 complete [ 2431.466613] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2440.376144] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2450.328774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2457.148063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2463.860770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2470.963112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2478.496983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2478.577397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.971560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2479.010425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2479.080302] Lustre: lustre-MDT0000: new disk, initializing [ 2479.171138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.184241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2483.362719] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2493.728816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2493.796833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2493.899716] Lustre: 149436:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2493.980835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2493.983912] Lustre: Skipped 1 previous similar message [ 2494.070974] Lustre: lustre-MDT0001: new disk, initializing [ 2494.166724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2494.204471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2494.209696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2497.839310] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2504.682773] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2507.997184] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2514.493692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2514.543644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2514.799903] Lustre: lustre-OST0000: new disk, initializing [ 2514.807677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2514.923872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.399363] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2524.705347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2524.709428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2524.814204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2526.792732] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2530.461778] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2530.746839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2534.912994] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2535.167261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2540.004175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.008402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2540.009752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.009758] Lustre: Skipped 2 previous similar messages [ 2540.034970] Lustre: Skipped 2 previous similar messages [ 2543.114093] LustreError: 152348:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2543.123761] LustreError: 152348:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2543.309908] Lustre: server umount lustre-OST0000 complete [ 2550.243130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.244767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.271723] Lustre: Skipped 1 previous similar message [ 2550.290928] Lustre: Skipped 2 previous similar messages [ 2552.895960] LustreError: 152550:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2552.902152] LustreError: 152550:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2555.065163] Lustre: server umount lustre-MDT0000 complete [ 2555.376944] LustreError: 150335:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.421565] LustreError: 150335:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 2559.200418] LustreError: 150213:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755041928 with bad export cookie 18367158321801096441 [ 2559.204734] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.218540] LustreError: 150213:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2559.663910] Lustre: server umount lustre-MDT0001 complete [ 2563.278286] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:38:51 (1755041931) [ 2564.984497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2566.777418] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 19:38:54 (1755041934) [ 2573.969610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.515934] LustreError: 153425:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.608989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.522857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2579.951062] LustreError: 153425:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.042314] LustreError: 153424:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.398302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2592.146737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2598.136417] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2602.231142] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2609.439258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.991392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.006362] Lustre: Skipped 1 previous similar message [ 2615.107348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2621.824476] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2629.603165] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.634385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.644405] Lustre: Skipped 1 previous similar message [ 2634.215423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2634.606686] LustreError: 155653:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2634.609184] LustreError: 155653:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2634.723432] LustreError: 153421:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.788804] LustreError: 153421:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2634.954885] Lustre: server umount lustre-MDT0000 complete [ 2640.033919] LustreError: 154140:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042009 with bad export cookie 18367158321801097323 [ 2640.040352] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.052966] LustreError: 154140:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2643.394068] LustreError: 153419:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.430056] LustreError: 153419:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2643.446239] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 2643.449201] Lustre: Skipped 2 previous similar messages [ 2644.969787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.991806] Lustre: Skipped 2 previous similar messages [ 2647.122259] Lustre: server umount lustre-MDT0001 complete [ 2655.684965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2671.342864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2671.894158] LustreError: 156916:0:(llog.c:1602:llog_backup()) MGC192.168.202.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2671.897800] Lustre: 156916:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2673.219426] LustreError: 156924:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.256937] LustreError: 156924:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2705.346234] LustreError: 156923:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.384696] LustreError: 156923:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 22 previous similar messages [ 2732.511404] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2732.517749] Lustre: 156916:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2759.647410] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2759.653253] Lustre: 153416:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2769.898150] LustreError: 156923:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.943052] LustreError: 156923:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 40 previous similar messages [ 2773.495873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2776.866621] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2778.604973] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 2780.127289] LustreError: 153416:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2784.240624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.753644] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x0 to 0xfee5412e285089b6 [ 2784.767615] Lustre: MGC192.168.202.116@tcp: Connection restored to (at 0@lo) [ 2788.627352] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2790.386788] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2795.766262] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.407908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2804.327373] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2806.930257] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2831.330471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2831.335640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.337968] Lustre: Skipped 3 previous similar messages [ 2831.343534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.406359] Lustre: Skipped 2 previous similar messages [ 2845.151130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2845.251672] LustreError: 158413:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2845.259661] LustreError: 158413:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2845.323037] Lustre: server umount lustre-MDT0000 complete [ 2848.172284] LustreError: 154140:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042217 with bad export cookie 18367158321801103798 [ 2848.175393] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2848.189797] LustreError: 154140:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2848.451679] Lustre: server umount lustre-MDT0001 complete [ 2860.948402] Lustre: server umount lustre-OST0000 complete [ 2877.079456] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2884.297830] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2890.768932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2896.007825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2900.666699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2904.955781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2909.060213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2909.091134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.207388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2909.228512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2909.271684] Lustre: lustre-MDT0000: new disk, initializing [ 2909.327318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.331836] Lustre: Skipped 1 previous similar message [ 2909.342244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2911.774461] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2918.657043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2918.707690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2918.785949] Lustre: 161748:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2918.954467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2921.617171] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2926.584498] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2928.921578] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2933.601339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2933.679956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2938.301186] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2941.480884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2941.487618] Lustre: Skipped 1 previous similar message [ 2941.492883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2941.544237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2943.655924] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2946.571369] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2946.701656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2949.621194] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2949.745481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2951.650028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.650376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2951.651107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.651111] Lustre: Skipped 7 previous similar messages [ 2951.656056] Lustre: Skipped 1 previous similar message [ 2957.055308] LustreError: 164653:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2957.064518] LustreError: 164653:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2957.176806] Lustre: server umount lustre-OST0000 complete [ 2966.630711] Lustre: server umount lustre-MDT0000 complete [ 2967.008560] LustreError: 161759:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.024422] LustreError: 161759:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 19 previous similar messages [ 2969.188047] LustreError: 163481:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042338 with bad export cookie 18367158321801120612 [ 2969.193306] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2969.197302] LustreError: 163481:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2969.476069] Lustre: server umount lustre-MDT0001 complete [ 2975.248370] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:45:43 (1755042343) [ 2979.817412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.661194] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2988.404343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2991.337138] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2995.663581] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2998.309564] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3002.205360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3006.370528] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3010.095162] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3020.485358] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:46:28 (1755042388) [ 3023.845026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.845725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3023.863505] Lustre: Skipped 2 previous similar messages [ 3023.875791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3023.878857] Lustre: Skipped 6 previous similar messages [ 3029.394329] LustreError: 168147:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3029.399401] LustreError: 168147:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3029.590624] Lustre: server umount lustre-OST0000 complete [ 3038.846650] Lustre: server umount lustre-MDT0000 complete [ 3041.485666] LustreError: 165707:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042410 with bad export cookie 18367158321801121494 [ 3041.487154] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.501152] LustreError: 165707:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3041.786521] Lustre: server umount lustre-MDT0001 complete [ 3050.490785] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3052.578681] Key type lgssc unregistered [ 3052.772646] LNet: 169078:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3053.798417] LNet: Removed LNI 192.168.202.116@tcp [ 3054.464869] Key type .llcrypt unregistered [ 3054.467129] Key type ._llcrypt unregistered [ 3071.217260] Key type ._llcrypt registered [ 3071.224926] Key type .llcrypt registered [ 3071.365967] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3072.859534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3072.879406] alg: No test for adler32 (adler32-zlib) [ 3074.171917] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 3074.465514] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3074.479340] LNet: Accept secure, port 988 [ 3076.154305] Key type lgssc registered [ 3077.413110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3083.697332] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3083.715335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3100.256997] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3106.399536] LustreError: 170292:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 3112.543606] LustreError: 170292:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3112.549350] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3112.552694] LustreError: MGC192.168.202.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3112.557904] LustreError: 170292:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3112.563311] LustreError: 170292:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -22 [ 3112.568134] LustreError: 170292:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 3112.574108] LustreError: 170292:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3112.628100] Lustre: server umount lustre-OST0000 complete [ 3112.630264] LustreError: 170292:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3116.938755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.230381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.514730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3124.693629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3124.855282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3126.484378] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3128.703052] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3130.092440] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3133.275533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3133.393126] LustreError: 172373:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3133.399552] Lustre: 172373:0:(mgc_request_server.c:610:mgc_llog_local_copy()) MGC192.168.202.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3133.465324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.358554] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3138.937896] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3139.964051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.965235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.969065] Lustre: Skipped 2 previous similar messages [ 3145.186527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.189919] Lustre: Skipped 4 previous similar messages [ 3150.304850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.307668] Lustre: Skipped 2 previous similar messages [ 3154.399335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3154.450264] LustreError: 173013:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3154.454657] LustreError: 173013:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3154.486886] Lustre: server umount lustre-MDT0000 complete [ 3155.424460] LustreError: 170790:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.431089] LustreError: 170790:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3156.021184] LustreError: 170772:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042525 with bad export cookie 14244843220146310324 [ 3156.022613] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.029576] LustreError: 170772:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3156.095834] LustreError: 173219:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3156.098587] LustreError: 173219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3156.196084] Lustre: server umount lustre-MDT0001 complete [ 3167.921977] LustreError: 173420:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3167.924653] LustreError: 173420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3167.989088] Lustre: server umount lustre-OST0000 complete [ 3171.883248] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3176.075783] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3180.887660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3184.051934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3186.527586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3188.827958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.689103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.718241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.860904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3191.887553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3191.937736] Lustre: lustre-MDT0000: new disk, initializing [ 3191.978118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.989175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3193.726027] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3199.022302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.053652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3199.095562] Lustre: 175737:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3199.126278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3199.128536] Lustre: Skipped 1 previous similar message [ 3199.168355] Lustre: lustre-MDT0001: new disk, initializing [ 3199.210426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.227426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3199.231514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3200.816265] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3204.654534] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3206.440381] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3209.549135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3209.584132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3209.700675] Lustre: lustre-OST0000: new disk, initializing [ 3209.704425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3209.750988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.124147] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3215.374609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3215.378900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3215.433378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3215.652305] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3217.418207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3217.527848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.255905] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3219.347331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3220.450559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3220.451342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.453793] LustreError: Skipped 1 previous similar message [ 3220.461389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3222.198245] LustreError: 178648:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3222.201382] LustreError: 178648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3222.297418] Lustre: server umount lustre-OST0000 complete [ 3225.056487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3225.059437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.063633] Lustre: Skipped 1 previous similar message [ 3225.065962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.068530] Lustre: Skipped 1 previous similar message [ 3235.812153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.815104] Lustre: Skipped 4 previous similar messages [ 3238.367658] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.408965] LustreError: 178848:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3238.411852] LustreError: 178848:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3238.451324] Lustre: server umount lustre-MDT0000 complete [ 3239.905361] LustreError: 175750:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.913995] LustreError: 175750:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3240.201233] LustreError: 177473:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042609 with bad export cookie 14244843220146311017 [ 3240.203990] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.206241] LustreError: 177473:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.396327] Lustre: server umount lustre-MDT0001 complete [ 3244.323181] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:50:12 (1755042612) [ 3251.727760] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3252.954450] Key type lgssc unregistered [ 3253.107860] LNet: 180227:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.181458] LNet: Removed LNI 192.168.202.116@tcp [ 3254.556048] Key type .llcrypt unregistered [ 3254.557586] Key type ._llcrypt unregistered [ 3263.384149] Key type ._llcrypt registered [ 3263.385380] Key type .llcrypt registered [ 3263.431947] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3263.860532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3263.904472] alg: No test for adler32 (adler32-zlib) [ 3264.792108] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 3264.893847] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3264.897505] LNet: Accept secure, port 988 [ 3266.487248] Key type lgssc registered [ 3267.000117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3270.674677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3270.688801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.961558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.842330] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3278.218855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.405454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.460262] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3282.903091] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3284.659961] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3288.815291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3288.921387] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3288.922876] LustreError: 183017:0:(llog.c:1642:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3288.929115] Lustre: 183017:0:(mgc_request_server.c:607:mgc_llog_local_copy()) MGC192.168.202.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3288.991895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.690777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3294.957865] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3298.783963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3298.787570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.796591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.298164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.304897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.308826] Lustre: Skipped 1 previous similar message [ 3299.313644] Lustre: Skipped 1 previous similar message [ 3302.296470] LustreError: 183661:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3302.358837] Lustre: server umount lustre-MDT0000 complete [ 3303.393492] LustreError: 181392:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.403253] LustreError: 181392:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3304.530140] LustreError: 181374:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042673 with bad export cookie 16685106316963175704 [ 3304.533610] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3304.535318] LustreError: 181374:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.504270] LustreError: 182099:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cca90569 x1840295585009024/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3305.532686] LustreError: 183865:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3305.540883] LustreError: 183865:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3305.684877] Lustre: server umount lustre-MDT0001 complete [ 3318.143421] LustreError: 184065:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3318.145800] LustreError: 184065:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3318.229703] Lustre: server umount lustre-OST0000 complete [ 3322.915797] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3327.594646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3332.378571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3335.494932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3339.029520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3342.174655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3345.535082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3345.586288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.737828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3345.754551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3345.801612] Lustre: lustre-MDT0000: new disk, initializing [ 3345.847115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.858642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3347.640749] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3352.625848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.671073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3352.716696] Lustre: 186383:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3352.739304] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3352.741245] Lustre: Skipped 1 previous similar message [ 3352.743456] LustreError: 187145:0:(llog.c:1642:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3352.747416] LustreError: 187145:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3352.749409] Lustre: 187145:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3352.758872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3352.761667] Lustre: Skipped 1 previous similar message [ 3352.802988] Lustre: lustre-MDT0001: new disk, initializing [ 3352.846640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.863719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3352.868869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3354.487297] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3357.949690] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3359.378504] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3362.129570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3362.157839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3362.255640] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3362.257064] Lustre: Skipped 1 previous similar message [ 3362.258774] LustreError: 188093:0:(llog.c:1642:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3362.262249] LustreError: 188093:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3362.264969] Lustre: 188093:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3362.269481] Lustre: 188093:0:(mgc_request_server.c:630:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3362.279244] Lustre: lustre-OST0000: new disk, initializing [ 3362.281858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3362.316407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.030579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3364.034851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3364.078182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3364.526478] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3367.905617] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3369.554898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3369.642690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3371.262284] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3371.358343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3374.563894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3374.564585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.567834] LustreError: Skipped 1 previous similar message [ 3374.568517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3374.573461] Lustre: Skipped 1 previous similar message [ 3378.227248] LustreError: 189290:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3378.229536] LustreError: 189290:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3378.325585] Lustre: server umount lustre-OST0000 complete [ 3383.776574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3383.781923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.787271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.789279] Lustre: Skipped 1 previous similar message [ 3386.197989] LustreError: 189491:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3386.202129] LustreError: 189491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3386.235089] Lustre: server umount lustre-MDT0000 complete [ 3388.128759] LustreError: 187159:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042757 with bad export cookie 16685106316963176404 [ 3388.133549] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3388.135302] LustreError: 187159:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3388.306592] Lustre: server umount lustre-MDT0001 complete [ 3392.697903] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:52:41 (1755042761) [ 3399.780304] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3404.413682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.632478] LustreError: 191051:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.656472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.636648] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3409.889175] LustreError: 191051:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.297739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.576874] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3415.141991] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3419.105191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3422.581479] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3427.554699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3427.601707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3427.693749] Lustre: lustre-OST0001: new disk, initializing [ 3427.696968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3430.663990] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3431.732502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3431.749410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3431.788153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3437.650558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3441.526722] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3448.014929] Lustre: DEBUG MARKER: ADD OST3 [ 3451.293224] loop: module loaded [ 3451.342105] loop0: detected capacity change from 0 to 204800000 [ 3451.470659] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3451.475958] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3451.483730] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3451.577952] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.475343] loop0: detected capacity change from 0 to 204800000 [ 3457.533477] Lustre: DEBUG MARKER: STOP OST3 [ 3459.459183] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.511650] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3459.675185] Lustre: lustre-OST0003: new disk, initializing [ 3459.681818] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3459.763826] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3459.770329] Lustre: Skipped 3 previous similar messages [ 3461.241056] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3461.387332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3461.390530] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3461.421297] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3461.575749] Lustre: Failing over lustre-OST0003 [ 3461.626138] LustreError: 195445:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3461.636957] LustreError: 195445:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3461.695042] Lustre: server umount lustre-OST0003 complete [ 3463.269731] LustreError: 195400:0:(osp_precreate.c:1283:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3464.164678] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3464.532827] LustreError: 193739:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.729797] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.730348] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3466.738187] Lustre: Skipped 1 previous similar message [ 3469.762121] LustreError: 192490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.775993] LustreError: 192490:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3469.887803] Lustre: DEBUG MARKER: START OST3 again [ 3475.050758] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3475.283658] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3476.457485] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3477.215466] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3477.224697] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 192.168.202.116@tcp (at 0@lo) [ 3479.545393] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3495.657921] Lustre: Failing over lustre-OST0003 [ 3495.690104] LustreError: 197460:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3495.697974] LustreError: 197460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3495.730968] Lustre: server umount lustre-OST0003 complete [ 3495.911758] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.915610] LustreError: 192492:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.918755] Lustre: Skipped 2 previous similar messages [ 3495.931983] LustreError: 192492:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3511.272040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3511.277023] Lustre: Skipped 5 previous similar messages [ 3516.067378] Lustre: server umount lustre-MDT0000 complete [ 3516.388608] LustreError: 191045:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.398728] LustreError: 191045:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [ 3518.101530] LustreError: 191031:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042887 with bad export cookie 16685106316963177272 [ 3518.103028] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.105524] LustreError: 191031:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3521.505487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.506850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3521.518212] Lustre: Skipped 5 previous similar messages [ 3524.361893] Lustre: server umount lustre-MDT0001 complete [ 3526.403367] Lustre: server umount lustre-OST0000 complete [ 3528.399736] LustreError: 198336:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3528.405391] LustreError: 198336:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 3528.555430] Lustre: server umount lustre-OST0001 complete [ 3531.771861] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3535.809635] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3541.453632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.975812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3547.942799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.029887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.451054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.489954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.627979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3554.649684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3554.693814] Lustre: lustre-MDT0000: new disk, initializing [ 3554.750291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3554.755818] Lustre: Skipped 1 previous similar message [ 3554.767028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3556.648869] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3562.343666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.392709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3562.447383] Lustre: 200501:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.611565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3564.467671] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3568.518488] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3570.284911] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3573.766952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3573.798274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3573.937771] Lustre: lustre-OST0000: new disk, initializing [ 3573.939331] Lustre: Skipped 1 previous similar message [ 3573.941809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3573.945251] Lustre: Skipped 2 previous similar messages [ 3575.129452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3575.134129] Lustre: Skipped 1 previous similar message [ 3575.141593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3575.208961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3576.751448] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3580.701683] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.575588] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3582.701929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3584.606756] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3584.708101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3585.508976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.512372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3585.516805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3585.520429] Lustre: Skipped 4 previous similar messages [ 3591.602656] Lustre: server umount lustre-OST0000 complete [ 3599.230210] LustreError: 203610:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3599.232214] LustreError: 203610:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3599.259185] Lustre: server umount lustre-MDT0000 complete [ 3600.569294] LustreError: 201278:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755042969 with bad export cookie 16685106316963179554 [ 3600.572543] LustreError: 201278:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3600.572909] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.714675] Lustre: server umount lustre-MDT0001 complete [ 3601.979357] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:56:10 (1755042970) [ 3607.312245] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3610.825733] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3614.502881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3617.219265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.602941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3622.674379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.799311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.832611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.929789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3625.944407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3625.985367] Lustre: lustre-MDT0000: new disk, initializing [ 3626.032063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3627.616153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3632.796632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3632.823199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3632.867308] Lustre: 206635:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3632.985876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3634.584372] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3638.404952] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3640.003714] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3643.376968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3643.407188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3645.204155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3646.362561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3650.192657] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3651.983858] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3652.083808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3653.883956] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3653.985856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3655.649291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3655.649804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.652984] LustreError: Skipped 3 previous similar messages [ 3655.654357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.660261] Lustre: Skipped 4 previous similar messages [ 3655.670501] Lustre: Skipped 6 previous similar messages [ 3660.957735] Lustre: server umount lustre-OST0000 complete [ 3669.156111] Lustre: server umount lustre-MDT0000 complete [ 3670.796783] LustreError: 206627:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043040 with bad export cookie 16685106316963180436 [ 3670.800577] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.803462] LustreError: 206627:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.957443] Lustre: server umount lustre-MDT0001 complete [ 3674.516673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.700057] LustreError: 210420:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.708622] LustreError: 210420:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3676.569187] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3681.005744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3683.055107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3685.560803] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3687.302762] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3690.568197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3690.761481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.764652] Lustre: Skipped 7 previous similar messages [ 3693.382177] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3695.612403] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3710.435146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.437956] LustreError: Skipped 1 previous similar message [ 3713.861360] Lustre: server umount lustre-OST0000 complete [ 3720.672144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.674681] Lustre: Skipped 10 previous similar messages [ 3721.556865] Lustre: server umount lustre-MDT0000 complete [ 3722.867075] LustreError: 210400:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043092 with bad export cookie 16685106316963181318 [ 3722.870426] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.871317] LustreError: 210400:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3723.010500] Lustre: server umount lustre-MDT0001 complete [ 3827.185692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3828.312866] Key type lgssc unregistered [ 3828.469432] LNet: 213727:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3829.541295] LNet: Removed LNI 192.168.202.116@tcp [ 3829.829091] Key type .llcrypt unregistered [ 3829.830733] Key type ._llcrypt unregistered [ 3837.786697] Key type ._llcrypt registered [ 3837.787858] Key type .llcrypt registered [ 3837.836236] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3844.075311] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3844.447360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3844.455407] alg: No test for adler32 (adler32-zlib) [ 3845.312460] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 3845.397532] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3845.399711] LNet: Accept secure, port 988 [ 3846.983159] Key type lgssc registered [ 3847.344962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3850.533994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3852.834549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3854.922808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3857.138427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3859.578127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3859.596022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3859.601657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.689790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3860.701699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3860.737440] Lustre: lustre-MDT0000: new disk, initializing [ 3860.768274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.776122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3862.109105] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3866.579976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3866.604141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3866.642670] Lustre: 217012:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.674935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.677792] Lustre: Skipped 1 previous similar message [ 3866.716467] Lustre: lustre-MDT0001: new disk, initializing [ 3866.750707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.765251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3866.769502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3868.317936] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3871.749284] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3872.995158] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3875.384563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3875.411867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3875.504715] Lustre: lustre-OST0000: new disk, initializing [ 3875.506788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3875.532323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.409296] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3879.610303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3879.613435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3879.649838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3880.598741] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3881.951435] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3882.027167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3883.391336] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3883.463666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3885.024544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.026061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.027584] LustreError: Skipped 1 previous similar message [ 3885.031058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.033453] Lustre: Skipped 1 previous similar message [ 3886.125315] LustreError: 219919:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3886.186201] Lustre: server umount lustre-OST0000 complete [ 3887.583933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3887.587110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.591438] Lustre: Skipped 1 previous similar message [ 3887.593883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.145056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.145458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.503369] LustreError: 220120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3893.505122] LustreError: 220120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3893.530720] Lustre: server umount lustre-MDT0000 complete [ 3894.766104] LustreError: 217785:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043264 with bad export cookie 3411529882825449518 [ 3894.768123] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3894.770210] LustreError: 217785:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3894.823065] LustreError: 220332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3894.825895] LustreError: 220332:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3894.903494] Lustre: server umount lustre-MDT0001 complete [ 3900.385541] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3903.660902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.824472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.215185] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3908.186470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3908.317181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3909.701767] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3910.739655] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3913.100530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3915.184730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3918.097709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3918.125079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3918.192694] Lustre: lustre-OST0001: new disk, initializing [ 3918.195105] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3918.238252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3918.239958] Lustre: Skipped 1 previous similar message [ 3920.208664] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3923.465600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3923.469243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3923.508152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3925.700350] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3928.307360] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3937.315966] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:01:46 (1755043306) [ 3938.787880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.789566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.792927] Lustre: Skipped 2 previous similar messages [ 3938.794362] Lustre: Skipped 2 previous similar messages [ 3943.907262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.908877] Lustre: Skipped 3 previous similar messages [ 3944.127890] LustreError: 224847:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3944.129895] LustreError: 224847:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3944.159935] Lustre: server umount lustre-MDT0000 complete [ 3945.389577] LustreError: 224565:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043314 with bad export cookie 3411529882825450386 [ 3945.390907] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3945.394230] LustreError: 224565:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3945.545712] Lustre: server umount lustre-MDT0001 complete [ 3957.235137] LustreError: 225250:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3957.237243] LustreError: 225250:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3957.253678] Lustre: server umount lustre-OST0000 complete [ 3967.729551] LustreError: 225453:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3967.731445] LustreError: 225453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3967.823923] Lustre: server umount lustre-OST0001 complete [ 3970.008958] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3973.473968] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3977.060424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3979.354041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3981.453808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3983.711765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3986.180881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3986.209746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.303280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3986.313695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3986.355473] Lustre: lustre-MDT0000: new disk, initializing [ 3986.382938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.389756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3987.775136] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3991.912186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3991.938287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3991.971814] Lustre: 227625:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3992.090891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3993.376242] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3996.508273] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3997.625763] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3999.851217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.874573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.971692] Lustre: lustre-OST0000: new disk, initializing [ 3999.972872] Lustre: Skipped 1 previous similar message [ 3999.974994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3999.976959] Lustre: Skipped 2 previous similar messages [ 4001.273849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4001.278340] Lustre: Skipped 1 previous similar message [ 4001.280221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4001.311988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4001.902458] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4005.014329] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4006.373236] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4006.443684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4007.771287] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4007.859112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4011.492976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.493140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.494843] LustreError: Skipped 1 previous similar message [ 4011.495841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.500782] Lustre: Skipped 2 previous similar messages [ 4011.501972] Lustre: Skipped 2 previous similar messages [ 4014.577082] LustreError: 230533:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4014.579429] LustreError: 230533:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4014.649307] Lustre: server umount lustre-OST0000 complete [ 4022.041133] Lustre: server umount lustre-MDT0000 complete [ 4023.331563] LustreError: 227616:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043392 with bad export cookie 3411529882825452486 [ 4023.335691] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.513943] Lustre: server umount lustre-MDT0001 complete [ 4026.414206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.580026] LustreError: 231410:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.585752] LustreError: 231410:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4026.606794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.608676] Lustre: Skipped 2 previous similar messages [ 4027.984962] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4030.862294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4032.354744] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4034.169850] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4035.383697] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4037.731170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4039.681659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4041.429754] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4185.570457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4185.570528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.572824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4185.575428] Lustre: Skipped 3 previous similar messages [ 4185.578129] Lustre: Skipped 6 previous similar messages [ 4191.404555] LustreError: 233677:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4191.406233] LustreError: 233677:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4191.468854] Lustre: server umount lustre-OST0000 complete [ 4194.783892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4207.071065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4207.146244] Lustre: server umount lustre-MDT0000 complete [ 4208.460508] LustreError: 231391:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043577 with bad export cookie 3411529882825453368 [ 4208.463868] LustreError: 231391:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4208.464512] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4208.600162] Lustre: server umount lustre-MDT0001 complete [ 4306.671550] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4307.667486] Key type lgssc unregistered [ 4307.802340] LNet: 234643:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4308.837386] LNet: Removed LNI 192.168.202.116@tcp [ 4309.138944] Key type .llcrypt unregistered [ 4309.139942] Key type ._llcrypt unregistered [ 4317.171129] Key type ._llcrypt registered [ 4317.172449] Key type .llcrypt registered [ 4317.208426] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4323.418196] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4323.809451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4323.822309] alg: No test for adler32 (adler32-zlib) [ 4324.671077] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 4324.753079] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4324.754498] LNet: Accept secure, port 988 [ 4326.335151] Key type lgssc registered [ 4326.686007] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4329.597206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4331.841016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4333.891765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4336.033181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4341.542252] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4344.877979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4344.897996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4344.903692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.985362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4345.996856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4346.036771] Lustre: lustre-MDT0000: new disk, initializing [ 4346.061648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.067223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4347.299767] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4351.257481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.283349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4351.312056] Lustre: 238614:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4351.332735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4351.334497] Lustre: Skipped 1 previous similar message [ 4351.371313] Lustre: lustre-MDT0001: new disk, initializing [ 4351.406564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4351.421893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4351.425195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4352.709392] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4354.963979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4357.197533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4357.221884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4357.321413] Lustre: lustre-OST0000: new disk, initializing [ 4357.324388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4357.355283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.840846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4358.843456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4358.889734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4359.197682] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4363.343622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4363.373510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4363.443952] Lustre: lustre-OST0001: new disk, initializing [ 4363.446616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4363.488579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4365.366634] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4365.650074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4365.655064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4365.689746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4370.752416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4374.410515] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4386.274044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.276253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.277520] Lustre: Skipped 2 previous similar messages [ 4386.279580] Lustre: Skipped 1 previous similar message [ 4388.382338] LustreError: 242115:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4388.426544] Lustre: server umount lustre-MDT0000 complete [ 4389.785255] LustreError: 238605:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043759 with bad export cookie 5402790216950959364 [ 4389.788741] LustreError: 238605:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4389.789711] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.858519] LustreError: 242316:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4389.860496] LustreError: 242316:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4389.939808] Lustre: server umount lustre-MDT0001 complete [ 4401.592162] LustreError: 242516:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4401.595557] LustreError: 242516:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4401.617262] Lustre: server umount lustre-OST0000 complete [ 4413.241917] LustreError: 242718:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4413.246272] LustreError: 242718:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4413.369533] Lustre: server umount lustre-OST0001 complete [ 4419.494814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4423.096113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4423.488056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4426.594086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4429.556170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4429.880563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4436.377074] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4439.673894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.679996] Lustre: lustre-MDT0000: reset Object Index mappings [ 4439.845288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.077643] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4444.390334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4444.398943] Lustre: lustre-MDT0001: reset Object Index mappings [ 4445.865451] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4447.122611] Lustre: 247113:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4450.128843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4450.285686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.287977] Lustre: Skipped 1 previous similar message [ 4452.507819] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4454.380167] LustreError: 247468:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.392101] LustreError: 247468:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4455.748351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4458.081084] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4465.387991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4465.388551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4472.170856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4473.640400] Lustre: 248967:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4480.479867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4480.482221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.485418] Lustre: Skipped 1 previous similar message [ 4480.487158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4480.488705] Lustre: Skipped 1 previous similar message [ 4486.112853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4486.115561] Lustre: Skipped 6 previous similar messages [ 4486.208761] LustreError: 249072:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4486.210867] LustreError: 249072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4486.239335] Lustre: server umount lustre-MDT0000 complete [ 4487.694201] LustreError: 245995:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043857 with bad export cookie 5402790216950966140 [ 4487.697778] LustreError: 245995:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4487.698351] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.839999] Lustre: server umount lustre-MDT0001 complete [ 4499.449168] LustreError: 249473:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4499.454471] LustreError: 249473:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4499.473961] Lustre: server umount lustre-OST0000 complete [ 4511.232658] Lustre: server umount lustre-OST0001 complete [ 4517.186390] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:11:26 (1755043886) [ 4522.657669] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4525.709101] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4529.716580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4532.386632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4534.702684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4536.951585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4539.184121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4539.205045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.279218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4539.289202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4539.325741] Lustre: lustre-MDT0000: new disk, initializing [ 4539.351236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4539.353214] Lustre: Skipped 1 previous similar message [ 4539.358696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4540.538539] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4544.484100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4544.505059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4544.532850] Lustre: 253177:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4544.628444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4545.905902] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4548.998161] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4550.174981] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4552.430896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4552.451756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4552.547194] Lustre: lustre-OST0000: new disk, initializing [ 4552.548398] Lustre: Skipped 1 previous similar message [ 4552.549904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4552.551699] Lustre: Skipped 2 previous similar messages [ 4553.727046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4553.732198] Lustre: Skipped 1 previous similar message [ 4553.735508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4553.772230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4554.480975] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4557.850462] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4559.474375] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4559.560061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4560.746603] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4560.824282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4563.628628] LustreError: 256084:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4563.630422] LustreError: 256084:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4563.690426] Lustre: server umount lustre-OST0000 complete [ 4569.056609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.057241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4569.059705] Lustre: Skipped 4 previous similar messages [ 4574.176059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4574.177816] Lustre: Skipped 2 previous similar messages [ 4579.295110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4579.348383] Lustre: server umount lustre-MDT0000 complete [ 4580.192068] Lustre: 236145:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755043933/real 1755043933] req@ffff939776d0a300 x1840296696613120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1755043949 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 4580.198128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.202053] LustreError: 253958:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.206365] LustreError: 253958:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4580.522343] LustreError: 253168:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043949 with bad export cookie 5402790216950968226 [ 4580.527129] LustreError: 253168:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4580.527581] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4580.670341] Lustre: server umount lustre-MDT0001 complete [ 4586.955927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4590.642364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.775159] LustreError: 257650:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.789172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.791458] Lustre: Skipped 2 previous similar messages [ 4592.105026] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4594.826507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4596.152138] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4597.103769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4599.196052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4601.144671] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4603.770942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.795271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4603.847731] Lustre: lustre-OST0001: new disk, initializing [ 4603.850657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4604.941399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4604.944263] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4604.980692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4605.640033] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4610.014530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4611.158579] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4618.191517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4618.196736] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4620.258895] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.262729] Lustre: Skipped 2 previous similar messages [ 4620.264243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.267474] Lustre: Skipped 5 previous similar messages [ 4626.301768] LustreError: 261129:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4626.304240] LustreError: 261129:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4626.329368] Lustre: server umount lustre-MDT0000 complete [ 4627.761351] LustreError: 258363:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755043997 with bad export cookie 5402790216950969115 [ 4627.768377] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4627.945878] Lustre: server umount lustre-MDT0001 complete [ 4638.657238] Lustre: server umount lustre-OST0000 complete [ 4650.378861] Lustre: server umount lustre-OST0001 complete [ 4655.904697] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4659.331297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.475132] LustreError: 262898:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.492924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.495101] Lustre: Skipped 3 previous similar messages [ 4660.900920] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4664.139586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4665.500929] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4666.534566] Lustre: 263994:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4668.983039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4670.119629] LustreError: 264350:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.126398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4671.290962] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4674.728439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4676.552733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4677.319550] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4681.084326] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4682.598687] Lustre: 265834:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4689.325876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4689.332363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4692.621791] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 20:14:21 (1755044061) [ 4695.009574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.014217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.020136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.023859] Lustre: Skipped 4 previous similar messages [ 4695.755812] LustreError: 266518:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4695.759237] LustreError: 266518:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4695.795691] Lustre: server umount lustre-MDT0000 complete [ 4697.123111] LustreError: 262879:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755044066 with bad export cookie 5402790216950975401 [ 4697.126249] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.127535] LustreError: 262879:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4699.311067] Lustre: server umount lustre-MDT0001 complete [ 4711.307757] Lustre: server umount lustre-OST0000 complete [ 4723.421598] Lustre: server umount lustre-OST0001 complete [ 4742.822205] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 20:15:11 (1755044111) [ 4757.901786] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4758.901210] Key type lgssc unregistered [ 4759.037396] LNet: 269520:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4760.101359] LNet: Removed LNI 192.168.202.116@tcp [ 4760.403174] Key type .llcrypt unregistered [ 4760.404158] Key type ._llcrypt unregistered [ 4788.588094] Key type ._llcrypt registered [ 4788.589517] Key type .llcrypt registered [ 4788.625348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4788.715323] Key type .llcrypt unregistered [ 4788.716663] Key type ._llcrypt unregistered [ 4800.304764] Key type ._llcrypt registered [ 4800.306065] Key type .llcrypt registered [ 4800.357171] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4800.476193] Key type .llcrypt unregistered [ 4800.477403] Key type ._llcrypt unregistered [ 4831.544604] Key type ._llcrypt registered [ 4831.546245] Key type .llcrypt registered [ 4831.595875] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4831.737597] Key type .llcrypt unregistered [ 4831.739344] Key type ._llcrypt unregistered [ 4843.901944] Key type ._llcrypt registered [ 4843.903828] Key type .llcrypt registered [ 4843.949513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4844.067505] Key type .llcrypt unregistered [ 4844.069358] Key type ._llcrypt unregistered [ 4856.665260] Key type ._llcrypt registered [ 4856.666650] Key type .llcrypt registered [ 4856.714836] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4856.878265] Key type .llcrypt unregistered [ 4856.879863] Key type ._llcrypt unregistered [ 4886.874148] Key type ._llcrypt registered [ 4886.875323] Key type .llcrypt registered [ 4886.916192] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4887.027363] Key type .llcrypt unregistered [ 4887.028699] Key type ._llcrypt unregistered [ 4898.699952] Key type ._llcrypt registered [ 4898.701189] Key type .llcrypt registered [ 4898.735792] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4898.848400] Key type .llcrypt unregistered [ 4898.849433] Key type ._llcrypt unregistered [ 4910.220216] Key type ._llcrypt registered [ 4910.221124] Key type .llcrypt registered [ 4910.260485] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4910.365344] Key type .llcrypt unregistered [ 4910.366507] Key type ._llcrypt unregistered [ 4930.481879] Key type ._llcrypt registered [ 4930.483769] Key type .llcrypt registered [ 4930.523932] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4936.914741] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4937.233287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.281743] alg: No test for adler32 (adler32-zlib) [ 4938.134744] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 4938.207721] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4938.209177] LNet: Accept secure, port 988 [ 4939.783242] Key type lgssc registered [ 4940.164535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4943.581486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4946.429305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4948.601641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4950.880928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4951.478132] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4813 sec ========== 20:18:40 (1755044320) [ 4952.172335] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:18:40 (1755044320) === [ 4953.599156] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:18:42 (1755044322) === [ 4968.073927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4969.371917] Key type lgssc unregistered [ 4969.506311] LNet: 285317:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4969.515303] LNet: Removed LNI 192.168.202.116@tcp [ 4969.883420] Key type .llcrypt unregistered [ 4969.884926] Key type ._llcrypt unregistered