[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489634273 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003305] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011071] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022812] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027518] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030227] *** VALIDATE tmpfs *** [ 0.032437] *** VALIDATE proc *** [ 0.034165] *** VALIDATE cgroup *** [ 0.035000] *** VALIDATE cgroup2 *** [ 0.035257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036180] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042175] debug: unmapping init [mem 0xffffffffb5a59000-0xffffffffb5a60fff] [ 0.044170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045662] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055417] smp: Bringing up secondary CPUs ... [ 0.056552] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.061091] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150198] node 0 deferred pages initialised in 83ms [ 0.153097] devtmpfs: initialized [ 0.154271] x86/mm: Memory block size: 128MB [ 0.157574] gcov: version magic: 0x41383552 [ 0.160087] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165316] pinctrl core: initialized pinctrl subsystem [ 0.168177] [ 0.168670] ************************************************************* [ 0.171014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ** ** [ 0.175011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177011] ** ** [ 0.179012] ** This means that this kernel is built to expose internal ** [ 0.180013] ** IOMMU data structures, which may compromise security on ** [ 0.184013] ** your system. ** [ 0.187010] ** ** [ 0.189010] ** If you see this message and you are not debugging the ** [ 0.192014] ** kernel, report this immediately to your vendor! ** [ 0.194012] ** ** [ 0.197014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199013] ************************************************************* [ 0.202745] NET: Registered protocol family 16 [ 0.204543] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213092] cpuidle: using governor menu [ 0.214811] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217375] PCI: Using configuration type 1 for base access [ 0.220210] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233087] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.237030] cryptd: max_cpu_qlen set to 1000 [ 0.239279] ACPI: Added _OSI(Module Device) [ 0.241028] ACPI: Added _OSI(Processor Device) [ 0.243016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249335] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255474] ACPI: Interpreter enabled [ 0.256000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256014] ACPI: Using IOAPIC for interrupt routing [ 0.257000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260538] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283300] acpiphp: Slot [2] registered [ 0.285108] acpiphp: Slot [5] registered [ 0.286109] acpiphp: Slot [6] registered [ 0.287097] acpiphp: Slot [7] registered [ 0.289095] acpiphp: Slot [8] registered [ 0.290106] acpiphp: Slot [9] registered [ 0.291091] acpiphp: Slot [10] registered [ 0.293110] acpiphp: Slot [3] registered [ 0.294067] acpiphp: Slot [4] registered [ 0.295083] acpiphp: Slot [11] registered [ 0.296102] acpiphp: Slot [12] registered [ 0.298082] acpiphp: Slot [13] registered [ 0.299077] acpiphp: Slot [14] registered [ 0.301140] acpiphp: Slot [15] registered [ 0.302098] acpiphp: Slot [16] registered [ 0.304095] acpiphp: Slot [17] registered [ 0.305061] acpiphp: Slot [18] registered [ 0.306089] acpiphp: Slot [19] registered [ 0.307201] acpiphp: Slot [20] registered [ 0.309101] acpiphp: Slot [21] registered [ 0.310124] acpiphp: Slot [22] registered [ 0.312085] acpiphp: Slot [23] registered [ 0.313100] acpiphp: Slot [24] registered [ 0.315111] acpiphp: Slot [25] registered [ 0.317100] acpiphp: Slot [26] registered [ 0.318096] acpiphp: Slot [27] registered [ 0.320220] acpiphp: Slot [28] registered [ 0.322107] acpiphp: Slot [29] registered [ 0.323139] acpiphp: Slot [30] registered [ 0.325096] acpiphp: Slot [31] registered [ 0.326065] PCI host bridge to bus 0000:00 [ 0.328020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.337026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.340026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348242] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357821] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363098] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373490] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376780] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382743] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409233] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452014] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.515018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.539019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553096] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.560015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593681] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.620017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.646879] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.656018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.686018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.699955] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.702442] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.705409] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.708377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.710218] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.715137] iommu: Default domain type: Passthrough [ 0.720447] SCSI subsystem initialized [ 0.724227] ACPI: bus type USB registered [ 0.726087] usbcore: registered new interface driver usbfs [ 0.728093] usbcore: registered new interface driver hub [ 0.729106] usbcore: registered new device driver usb [ 0.731130] pps_core: LinuxPPS API ver. 1 registered [ 0.733010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735076] PTP clock support registered [ 0.737113] EDAC MC: Ver: 3.0.0 [ 0.738509] PCI: Using ACPI for IRQ routing [ 0.739675] NetLabel: Initializing [ 0.740011] NetLabel: domain hash size = 128 [ 0.741011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.742167] NetLabel: unlabeled traffic allowed by default [ 0.744070] vgaarb: loaded [ 0.745320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.747014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.752000] clocksource: Switched to clocksource kvm-clock [ 0.852426] VFS: Disk quotas dquot_6.6.0 [ 0.854480] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857122] *** VALIDATE ramfs *** [ 0.858226] *** VALIDATE hugetlbfs *** [ 0.859469] pnp: PnP ACPI init [ 0.862095] pnp: PnP ACPI: found 6 devices [ 0.878380] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.881449] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.883367] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.885542] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.887403] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.889235] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891861] NET: Registered protocol family 2 [ 0.894388] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899249] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.902892] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.907399] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910668] TCP: Hash tables configured (established 65536 bind 65536) [ 0.913623] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.916729] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.919114] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.922453] NET: Registered protocol family 1 [ 0.925809] RPC: Registered named UNIX socket transport module. [ 0.928203] RPC: Registered udp transport module. [ 0.929985] RPC: Registered tcp transport module. [ 0.931884] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.934312] NET: Registered protocol family 44 [ 0.936153] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.938550] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.940886] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943530] PCI: CLS 0 bytes, default 64 [ 0.945360] Unpacking initramfs... [ 2.352913] debug: unmapping init [mem 0xffff8b763cc54000-0xffff8b763ffbffff] [ 2.357453] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.360055] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.363176] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.871361] Initialise system trusted keyrings [ 2.872994] Key type blacklist registered [ 2.874686] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.882523] zbud: loaded [ 2.885234] *** VALIDATE nfs *** [ 2.886973] *** VALIDATE nfs4 *** [ 2.888591] pstore: using deflate compression [ 2.891719] Platform Keyring initialized [ 3.000745] NET: Registered protocol family 38 [ 3.002583] Key type asymmetric registered [ 3.004145] Asymmetric key parser 'x509' registered [ 3.005964] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.009049] io scheduler mq-deadline registered [ 3.010899] io scheduler kyber registered [ 3.012584] io scheduler bfq registered [ 3.014699] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.017850] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.020923] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024047] ACPI: Power Button [PWRF] [ 3.029381] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.035861] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.054314] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.061073] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.082799] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.109743] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.137622] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.142671] Non-volatile memory driver v1.3 [ 3.144071] Linux agpgart interface v0.103 [ 3.173508] virtio_blk virtio1: [vda] 147224 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.176522] vda: detected capacity change from 0 to 75378688 [ 3.190803] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.193526] vdb: detected capacity change from 0 to 1073741824 [ 3.212523] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.216424] vdc: detected capacity change from 0 to 2621440000 [ 3.231233] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.234129] vdd: detected capacity change from 0 to 2621440000 [ 3.250477] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.253364] vde: detected capacity change from 0 to 4294967296 [ 3.270342] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.273714] vdf: detected capacity change from 0 to 4294967296 [ 3.283081] libphy: Fixed MDIO Bus: probed [ 3.291436] usbcore: registered new interface driver usbserial_generic [ 3.293437] usbserial: USB Serial support registered for generic [ 3.295413] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.299481] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.301433] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.304299] mousedev: PS/2 mouse device common for all mice [ 3.307148] rtc_cmos 00:05: RTC can wake from S4 [ 3.310206] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.311233] rtc_cmos 00:05: registered as rtc0 [ 3.316304] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.319232] intel_pstate: CPU model not supported [ 3.321782] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.325856] hid: raw HID events driver (C) Jiri Kosina [ 3.327784] usbcore: registered new interface driver usbhid [ 3.328074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.329934] usbhid: USB HID core driver [ 3.335747] drop_monitor: Initializing network drop monitor service [ 3.338328] Initializing XFRM netlink socket [ 3.340398] NET: Registered protocol family 10 [ 3.344123] Segment Routing with IPv6 [ 3.345624] NET: Registered protocol family 17 [ 3.348133] mpls_gso: MPLS GSO support [ 3.355268] RAS: Correctable Errors collector initialized. [ 3.357543] AVX version of gcm_enc/dec engaged. [ 3.359360] AES CTR mode by8 optimization enabled [ 3.440713] sched_clock: Marking stable (3440679490, 0)->(4367740659, -927061169) [ 3.445368] registered taskstats version 1 [ 3.448223] Loading compiled-in X.509 certificates [ 3.450530] zswap: loaded using pool lzo/zbud [ 3.482995] Key type big_key registered [ 3.500197] Key type encrypted registered [ 3.501705] ima: No TPM chip found, activating TPM-bypass! [ 3.504075] ima: Allocated hash algorithm: sha1 [ 3.505604] ima: No architecture policies found [ 3.507087] evm: Initialising EVM extended attributes: [ 3.508787] evm: security.selinux [ 3.510106] evm: security.ima [ 3.511225] evm: security.capability [ 3.512422] evm: HMAC attrs: 0x1 [ 3.516168] rtc_cmos 00:05: setting system clock to 2026-09-11 15:13:11 UTC (1789139591) [ 3.522128] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.525410] debug: unmapping init [mem 0xffffffffb5782000-0xffffffffb5a58fff] [ 3.534107] Write protecting the kernel read-only data: 28672k [ 3.537467] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.540785] debug: unmapping init [mem 0xffffffffb4714000-0xffffffffb47fffff] [ 3.582041] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.590715] systemd[1]: Detected virtualization kvm. [ 3.592496] systemd[1]: Detected architecture x86-64. [ 3.594245] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.623312] systemd[1]: No hostname configured. [ 3.625087] systemd[1]: Set hostname to . [ 3.627060] random: systemd: uninitialized urandom read (16 bytes read) [ 3.629364] systemd[1]: Initializing machine ID from random generator. [ 3.685363] random: ln: uninitialized urandom read (6 bytes read) [ 3.772539] random: systemd: uninitialized urandom read (16 bytes read) [ 3.775599] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.781795] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.787837] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.492203] device-mapper: uevent: version 1.0.3 [ 4.494589] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.272824] virtio_net virtio0 ens2: renamed from eth0 [ 5.313513] random: fast init done [ 5.394230] scsi host0: ata_piix [ 5.430255] scsi host1: ata_piix [ 5.434167] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.437342] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.810263] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.019107] random: crng init done [ 10.020523] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.643372] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.809945] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.077891] SELinux: Disabled at runtime. [ 12.145375] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.153708] systemd[1]: Detected virtualization kvm. [ 12.155759] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.732357] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.735888] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.745502] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.749129] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.751836] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.759872] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.785303] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ 12.846411] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.268532] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.528253] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.545524] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.720337] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.738071] EDAC sbridge: Ver: 1.1.2 [ 15.596955] Key type dns_resolver registered [ 15.923322] NFS: Registering the id_resolver key type [ 15.926244] Key type id_resolver registered [ 15.928591] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg414-server login: [ 43.536442] libcfs: loading out-of-tree module taints kernel. [ 43.557339] Key type ._llcrypt registered [ 43.558766] Key type .llcrypt registered [ 43.610696] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 59.476182] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 61.390395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.415484] alg: No test for adler32 (adler32-zlib) [ 63.057177] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 64.138307] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 65.960610] Key type lgssc registered [ 67.768727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.092603] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.413043] hrtimer: interrupt took 4433961 ns [ 131.424515] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 145.121430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.176525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.600270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.687869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.860463] Lustre: lustre-MDT0000: new disk, initializing [ 147.041748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.066601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.802727] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.690394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.851544] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.900962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.908111] Lustre: Skipped 1 previous similar message [ 170.012595] Lustre: lustre-MDT0001: new disk, initializing [ 170.090759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.122861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.142996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.698209] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.567121] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.804518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.221882] Lustre: lustre-OST0000: new disk, initializing [ 192.226645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.231414] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.314325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.212232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.225366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.295433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.868048] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.761207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.919676] Lustre: lustre-OST0001: new disk, initializing [ 215.931364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.938830] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.024764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.319470] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.817239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.838861] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.967120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.581539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.815815] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 251.141653] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.289412] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 262.549950] Lustre: DEBUG MARKER: Client: 2.17.57.137 [ 265.768628] Lustre: DEBUG MARKER: MDS: 2.17.57.137 [ 268.495626] Lustre: DEBUG MARKER: OSS: 2.17.57.137 [ 270.338149] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 11:17:37 EDT 2026 [ 289.523975] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 291.706978] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 311.270428] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.272627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.296351] Lustre: Skipped 1 previous similar message [ 311.327025] Lustre: Skipped 3 previous similar messages [ 314.843087] Lustre: server umount lustre-MDT0000 complete [ 319.048453] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139907 with bad export cookie 2597773297320289302 [ 319.058250] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 319.062744] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.520555] LustreError: 8623:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.525772] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.545131] LustreError: 8623:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 321.571776] Lustre: Skipped 2 previous similar messages [ 321.582487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.592354] Lustre: Skipped 1 previous similar message [ 324.515245] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.519119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.551556] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 324.573448] Lustre: Skipped 1 previous similar message [ 325.506164] Lustre: server umount lustre-MDT0001 complete [ 329.507801] Lustre: server umount lustre-OST0000 complete [ 333.623890] Lustre: server umount lustre-OST0001 complete [ 340.490283] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 350.361346] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 389.524809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.836239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.873905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 389.941946] Lustre: lustre-MDT0000: new disk, initializing [ 390.038414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.074932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.763433] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 407.951286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.070492] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.149304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.154909] Lustre: Skipped 1 previous similar message [ 408.264885] Lustre: lustre-MDT0001: new disk, initializing [ 408.453657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.497311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.514790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.564239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 423.697451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 431.207684] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 440.177831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.615189] Lustre: lustre-OST0000: new disk, initializing [ 440.626871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.642722] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.757953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.450073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.471685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.609329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.103808] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 459.932606] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.056421] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.273355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.446978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.732972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 478.691157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.696433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.704614] LustreError: Skipped 1 previous similar message [ 478.711047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.731387] Lustre: Skipped 2 previous similar messages [ 480.055549] Lustre: server umount lustre-OST0000 complete [ 485.345540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 485.354155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.378934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.382852] Lustre: Skipped 1 previous similar message [ 490.440761] Lustre: server umount lustre-MDT0000 complete [ 494.058630] LustreError: 15092:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.087307] LustreError: 15092:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 495.269087] LustreError: 17049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140083 with bad export cookie 2597773297320290835 [ 495.272881] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.296121] LustreError: 17049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 495.954105] Lustre: server umount lustre-MDT0001 complete [ 503.517231] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:21:29 (1789140089) [ 524.310135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.927327] LustreError: 19928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.034552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.403987] LustreError: 19929:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.414381] LustreError: 19929:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 531.300273] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 539.619752] LustreError: 19928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.648889] LustreError: 19928:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 541.133291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.540908] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 557.279451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 563.837416] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 565.856134] Lustre: Failing over lustre-MDT0000 [ 566.081036] Lustre: server umount lustre-MDT0000 complete [ 566.882331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.907810] Lustre: Skipped 1 previous similar message [ 566.931892] LustreError: 19928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.974575] LustreError: 19928:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 582.625945] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140154/real 1789140154] req@ffff8b758829e580 x1876048899902720/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789140170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 582.669868] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.332370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.295354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.304474] Lustre: Skipped 1 previous similar message [ 599.529068] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 599.533850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 599.562416] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 599.716671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 610.422996] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 614.883384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.900902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.924893] Lustre: Skipped 1 previous similar message [ 614.933565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.959306] Lustre: Skipped 2 previous similar messages [ 620.687441] Lustre: server umount lustre-MDT0000 complete [ 622.055674] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.073707] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 624.926921] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140212 with bad export cookie 2597773297320292249 [ 624.935577] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.942772] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 625.417975] Lustre: server umount lustre-MDT0001 complete [ 635.418869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.795766] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 651.167278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.398935] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 667.731610] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 677.231397] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 680.804931] Lustre: Failing over lustre-MDT0000 [ 681.375763] Lustre: server umount lustre-MDT0000 complete [ 682.469581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.475734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.490417] Lustre: Skipped 1 previous similar message [ 690.665103] LustreError: 23273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.711687] LustreError: 23273:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 701.922580] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140273/real 1789140273] req@ffff8b76b76a0000 x1876048899935488/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789140289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 701.974592] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.730955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.140663] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x240d250ce1c57343 [ 711.152186] Lustre: MGC192.168.204.114@tcp: Connection restored to 0@lo (at 0@lo) [ 711.160245] Lustre: Skipped 1 previous similar message [ 711.415404] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.382655] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 716.775553] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 716.780599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 716.828362] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 727.935200] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 737.254954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 737.270860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.291112] Lustre: Skipped 1 previous similar message [ 737.298655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 737.302969] Lustre: Skipped 2 previous similar messages [ 739.307765] Lustre: server umount lustre-MDT0000 complete [ 742.966690] LustreError: 25939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140330 with bad export cookie 2597773297320293187 [ 742.975154] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.980192] LustreError: 25939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 743.255892] Lustre: server umount lustre-MDT0001 complete [ 752.333072] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:25:39 (1789140339) [ 769.288621] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 779.523718] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 826.253409] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 840.318766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.567331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 840.603885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 840.685370] Lustre: lustre-MDT0000: new disk, initializing [ 840.808110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.816914] Lustre: Skipped 3 previous similar messages [ 840.852304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 846.307158] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 858.486425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.610963] Lustre: 29652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 859.024880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 863.982813] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 874.140409] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.695688] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 888.005112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 888.293761] Lustre: 31617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 890.101387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 890.152049] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 890.231527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 890.269428] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 895.872401] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 906.227563] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.944323] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 936.422639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.425028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.451806] Lustre: Skipped 1 previous similar message [ 936.467709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.469029] LustreError: Skipped 1 previous similar message [ 936.488586] Lustre: Skipped 3 previous similar messages [ 950.240827] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 950.523485] Lustre: server umount lustre-OST0000 complete [ 956.386187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 960.699680] Lustre: server umount lustre-MDT0000 complete [ 962.018473] LustreError: 29661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.055210] LustreError: 29661:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 965.128707] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140553 with bad export cookie 2597773297320293586 [ 965.129241] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.145823] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 965.681362] Lustre: server umount lustre-MDT0001 complete [ 980.468511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 984.721259] Key type lgssc unregistered [ 984.971709] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.984024] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.998435] LNet: Removed LNI 192.168.204.114@tcp [ 986.614167] Key type .llcrypt unregistered [ 986.619387] Key type ._llcrypt unregistered [ 1021.056280] Key type ._llcrypt registered [ 1021.066444] Key type .llcrypt registered [ 1021.205947] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1022.420167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.464343] alg: No test for adler32 (adler32-zlib) [ 1023.701991] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1024.064750] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1025.801642] Key type lgssc registered [ 1027.418690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1036.859664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1036.886259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.360197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.182425] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1048.552359] LustreError: 35023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.575967] LustreError: 35023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1053.670811] LustreError: 35024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.725840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.236824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1060.519963] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1069.488667] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1077.568675] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1085.664750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1086.177585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.045492] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1100.556923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1102.061377] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1108.838702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1126.888557] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:31:53 (1789140713) [ 1129.861416] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1132.589335] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1142.338765] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:32:08 (1789140728) [ 1152.517837] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:32:19 (1789140739) [ 1153.932306] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1164.568668] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1164.581092] Lustre: Skipped 2 previous similar messages [ 1166.823084] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:32:33 (1789140753) [ 1169.279732] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1169.289038] Lustre: Skipped 2 previous similar messages [ 1178.970122] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1178.974557] Lustre: Skipped 4 previous similar messages [ 1195.297386] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:33:02 (1789140782) [ 1197.991985] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1201.503312] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1211.513296] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:33:18 (1789140798) [ 1213.426513] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1213.435557] Lustre: Skipped 6 previous similar messages [ 1229.389676] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:33:35 (1789140815) [ 1249.250453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1249.265149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.308807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1252.338865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1252.349257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1252.357819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.816586] Lustre: server umount lustre-OST0000 complete [ 1260.003256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1260.011992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.025503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1260.030678] Lustre: Skipped 1 previous similar message [ 1262.578634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.588464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.616033] Lustre: Skipped 1 previous similar message [ 1267.685551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.706947] Lustre: Skipped 1 previous similar message [ 1273.314111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1273.578453] Lustre: server umount lustre-MDT0000 complete [ 1277.781800] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140865 with bad export cookie 12582343551325789733 [ 1277.785297] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.791407] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1278.157946] Lustre: server umount lustre-MDT0001 complete [ 1291.707771] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1295.327205] Key type lgssc unregistered [ 1295.693296] LNet: 42905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.705711] LNetError: 42905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.738963] LNet: Removed LNI 192.168.204.114@tcp [ 1296.745595] Key type .llcrypt unregistered [ 1296.760732] Key type ._llcrypt unregistered [ 1318.144685] Key type ._llcrypt registered [ 1318.147401] Key type .llcrypt registered [ 1318.251362] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1336.810757] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1338.752793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.960761] alg: No test for adler32 (adler32-zlib) [ 1340.025171] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1340.225817] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1341.922128] Key type lgssc registered [ 1343.637338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1387.109927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1387.170115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.525580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1388.570208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1388.679879] Lustre: lustre-MDT0000: new disk, initializing [ 1388.772057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.785228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1393.690305] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1404.794973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.944365] Lustre: 46065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1404.992931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1404.996711] Lustre: Skipped 1 previous similar message [ 1405.055271] Lustre: lustre-MDT0001: new disk, initializing [ 1405.135394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1405.172171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1405.179671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1410.445026] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1420.857082] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1427.036670] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1434.156668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.479695] Lustre: lustre-OST0000: new disk, initializing [ 1434.486952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1434.497745] Lustre: 48035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1434.614047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.270065] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1443.888382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1443.896736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1444.024575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1451.163523] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1456.110525] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1456.316615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1461.036372] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1461.212456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1464.808501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1464.820772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.828273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1464.840530] LustreError: Skipped 1 previous similar message [ 1464.853959] Lustre: Skipped 1 previous similar message [ 1469.535084] Lustre: server umount lustre-OST0000 complete [ 1475.044934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.047510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.073672] Lustre: Skipped 2 previous similar messages [ 1475.103441] Lustre: Skipped 1 previous similar message [ 1480.166126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.179112] Lustre: Skipped 1 previous similar message [ 1480.733561] Lustre: server umount lustre-MDT0000 complete [ 1486.749578] LustreError: 46057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141074 with bad export cookie 4546508751255714343 [ 1486.751916] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.763673] LustreError: 46057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1486.989253] Lustre: server umount lustre-MDT0001 complete [ 1501.170716] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:38:07 (1789141087) [ 1508.768226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.285745] LustreError: 50494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.304880] LustreError: 50494:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1509.381101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.384632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1514.482954] LustreError: 50495:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.588127] LustreError: 50494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.677389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.950847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.358925] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1541.820836] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1549.266619] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1558.224627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.660902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.999878] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1575.234358] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1585.769322] LustreError: 53023:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1586.671663] Lustre: Modifying parameter general.timeout=1 in log params [ 1588.066106] Lustre: Modifying parameter general.timeout=2 in log params [ 1590.655911] Lustre: Modifying parameter general.timeout=5 in log params [ 1590.668034] Lustre: Skipped 2 previous similar messages [ 1594.679659] Lustre: Modifying parameter general.timeout=9 in log params [ 1594.684207] Lustre: Skipped 3 previous similar messages [ 1603.387813] Lustre: Modifying parameter general.timeout=18 in log params [ 1603.391234] Lustre: Skipped 8 previous similar messages [ 1620.046824] Lustre: Modifying parameter general.timeout=34 in log params [ 1620.057358] Lustre: Skipped 15 previous similar messages [ 1652.145398] Lustre: Modifying parameter general.timeout=63 in log params [ 1652.151436] Lustre: Skipped 28 previous similar messages [ 1717.545414] Lustre: Modifying parameter general.timeout=124 in log params [ 1717.549959] Lustre: Skipped 60 previous similar messages [ 1739.481324] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:42:06 (1789141326) [ 1742.611282] Lustre: 59979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1744.762831] Lustre: 60078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1746.928970] Lustre: 60190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1751.610348] Lustre: 60386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1751.627785] Lustre: 60386:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1785.192681] Lustre: 61858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1785.200603] Lustre: 61858:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1785.232414] LustreError: 61858:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1790.109236] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:42:56 (1789141376) [ 1806.305939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.325777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.334884] Lustre: Skipped 2 previous similar messages [ 1806.347958] Lustre: Skipped 2 previous similar messages [ 1811.034986] Lustre: server umount lustre-MDT0000 complete [ 1811.429871] LustreError: 50495:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.446414] LustreError: 50495:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1815.484088] LustreError: 50475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141403 with bad export cookie 4546508751255715225 [ 1815.485196] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.504305] LustreError: 50475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1815.873454] Lustre: server umount lustre-MDT0001 complete [ 1829.293260] Lustre: server umount lustre-OST0000 complete [ 1861.131484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.467975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1861.511981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1861.537453] Lustre: Skipped 28 previous similar messages [ 1861.982142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.504808] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1878.306392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.423385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1878.457732] Lustre: 64393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.613966] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1894.598477] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1903.522647] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1912.698561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1912.865645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1913.357312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.364146] Lustre: Skipped 1 previous similar message [ 1921.866508] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1932.137347] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1934.566692] Lustre: 66958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1944.034288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1944.054634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.089901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.466880] Lustre: server umount lustre-OST0000 complete [ 1954.792172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.797912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.816027] Lustre: Skipped 2 previous similar messages [ 1954.851514] Lustre: Skipped 2 previous similar messages [ 1959.161783] Lustre: server umount lustre-MDT0000 complete [ 1959.908158] LustreError: 64398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.939577] LustreError: 64398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1963.466756] LustreError: 66251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141551 with bad export cookie 4546508751255718368 [ 1963.479157] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.499381] LustreError: 66251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1963.917053] Lustre: server umount lustre-MDT0001 complete [ 1977.431871] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1980.552872] Key type lgssc unregistered [ 1980.916508] LNet: 68170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.921342] LNetError: 68170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.935315] LNet: Removed LNI 192.168.204.114@tcp [ 1981.792088] Key type .llcrypt unregistered [ 1981.796565] Key type ._llcrypt unregistered [ 2012.532707] Key type ._llcrypt registered [ 2012.536160] Key type .llcrypt registered [ 2012.661965] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2014.090194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2014.274981] alg: No test for adler32 (adler32-zlib) [ 2015.314512] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2015.641303] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2017.352247] Key type lgssc registered [ 2018.682820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2027.261407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2027.276776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.941709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.434495] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2038.257947] LustreError: 69591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.274734] LustreError: 69591:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2043.365818] LustreError: 69590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.215568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.860598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.208157] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2061.140423] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.648171] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2077.581978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.256688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.183228] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2096.337455] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.422783] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2123.233480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.242269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.270352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.270076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.271963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.291923] Lustre: Skipped 1 previous similar message [ 2124.304995] Lustre: Skipped 2 previous similar messages [ 2128.004148] Lustre: server umount lustre-MDT0000 complete [ 2129.379668] LustreError: 69591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.405899] LustreError: 69591:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2132.292553] LustreError: 72632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141720 with bad export cookie 1922597939496266294 [ 2132.294939] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.309658] LustreError: 72632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2132.941825] Lustre: server umount lustre-MDT0001 complete [ 2148.522511] Lustre: server umount lustre-OST0000 complete [ 2180.018115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.494551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2180.548831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2180.762613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.046572] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2196.698452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.797709] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2196.828853] Lustre: 74460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2196.887259] LustreError: 74471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.238459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2202.598823] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2212.912423] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2219.794386] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.467083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2227.695454] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2228.259412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.709066] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2242.566821] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2244.475795] Lustre: 77037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2246.187741] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2251.235489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2251.246526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.251358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2251.271433] Lustre: Skipped 1 previous similar message [ 2251.291326] Lustre: Skipped 1 previous similar message [ 2256.365502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.380051] Lustre: Skipped 1 previous similar message [ 2261.478742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2261.487548] Lustre: Skipped 1 previous similar message [ 2265.057099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2265.303255] Lustre: server umount lustre-OST0000 complete [ 2271.714018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.724213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.730366] Lustre: Skipped 1 previous similar message [ 2271.747258] Lustre: Skipped 1 previous similar message [ 2275.855114] Lustre: server umount lustre-MDT0000 complete [ 2276.837755] LustreError: 74467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.854345] LustreError: 74467:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2279.397996] LustreError: 74453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141867 with bad export cookie 1922597939496267141 [ 2279.403443] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.415430] LustreError: 74453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.709523] Lustre: server umount lustre-MDT0001 complete [ 2289.565145] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2292.353387] Key type lgssc unregistered [ 2292.664547] LNet: 78253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.669324] LNetError: 78253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.681399] LNet: Removed LNI 192.168.204.114@tcp [ 2293.356959] Key type .llcrypt unregistered [ 2293.361623] Key type ._llcrypt unregistered [ 2322.038853] Key type ._llcrypt registered [ 2322.046485] Key type .llcrypt registered [ 2322.206202] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2323.482202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.620187] alg: No test for adler32 (adler32-zlib) [ 2324.768654] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2325.077396] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2326.800232] Key type lgssc registered [ 2328.619794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.159809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2339.183642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.942507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.993370] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2350.051396] LustreError: 79685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.076203] LustreError: 79685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2355.171082] LustreError: 79686:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.899357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.951729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2365.263638] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2374.939280] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2381.387788] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2388.927687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2389.715093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.086494] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2403.311723] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.651816] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2415.735493] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2417.611283] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2417.622879] Lustre: Skipped 1 previous similar message [ 2420.592316] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2420.596168] Lustre: Skipped 2 previous similar messages [ 2425.051605] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2425.068491] Lustre: Skipped 3 previous similar messages [ 2434.139967] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2434.144590] Lustre: Skipped 7 previous similar messages [ 2450.693368] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2450.706703] Lustre: Skipped 14 previous similar messages [ 2483.405845] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2483.413997] Lustre: Skipped 28 previous similar messages [ 2527.843280] Lustre: 87793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2530.369185] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:55:17 (1789142117) [ 2532.162661] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2534.537218] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:55:21 (1789142121) [ 2538.473427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.484598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.484760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.495784] Lustre: Skipped 1 previous similar message [ 2543.587303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.596943] Lustre: Skipped 1 previous similar message [ 2544.709667] Lustre: server umount lustre-OST0000 complete [ 2548.705848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.715397] Lustre: Skipped 1 previous similar message [ 2548.719361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.318105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2553.333638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.350020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.557830] Lustre: server umount lustre-MDT0000 complete [ 2558.168158] LustreError: 79668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142146 with bad export cookie 7135897211235175086 [ 2558.168706] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.174827] LustreError: 79668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2558.482212] Lustre: server umount lustre-MDT0001 complete [ 2568.270819] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2570.775889] Key type lgssc unregistered [ 2571.057672] LNet: 89044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.066047] LNetError: 89044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2571.087342] LNet: Removed LNI 192.168.204.114@tcp [ 2572.013375] Key type .llcrypt unregistered [ 2572.016247] Key type ._llcrypt unregistered [ 2576.337976] Key type ._llcrypt registered [ 2576.340422] Key type .llcrypt registered [ 2576.448106] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2581.305214] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules [ 2582.331813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2582.392990] alg: No test for adler32 (adler32-zlib) [ 2583.569884] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2583.798064] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2585.514568] Key type lgssc registered [ 2586.881395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.201182] LustreError: 90248:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2607.087767] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2610.031173] LustreError: 90248:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2616.636515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2616.667390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.248110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.589894] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2628.588161] LustreError: 91250:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.611184] LustreError: 91250:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2631.329743] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:56:58 (1789142218) [ 2633.708821] LustreError: 91251:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.557746] LustreError: 91282:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2636.570879] LustreError: 91282:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2636.581469] LustreError: 91282:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2636.979806] Lustre: server umount lustre-MDT0000 complete [ 2652.536945] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2655.481414] Key type lgssc unregistered [ 2655.763863] LNet: 92782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2655.774909] LNetError: 92782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2655.791166] LNet: Removed LNI 192.168.204.114@tcp [ 2656.538851] Key type .llcrypt unregistered [ 2656.543282] Key type ._llcrypt unregistered [ 2680.803794] Key type ._llcrypt registered [ 2680.806756] Key type .llcrypt registered [ 2681.016743] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2683.141225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2683.201988] alg: No test for adler32 (adler32-zlib) [ 2684.610837] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2684.858270] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2686.544197] Key type lgssc registered [ 2687.740852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2697.735560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2697.776988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.425761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.346894] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2709.991493] LustreError: 94000:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.028540] LustreError: 94000:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2714.227646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.978959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2719.192455] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2727.759598] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2733.895537] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2741.605766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2742.163857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.178327] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2757.769268] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2808.407888] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:59:55 (1789142395) [ 2810.288738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2812.120691] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:59:59 (1789142399) [ 2816.995042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2817.010629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.031113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.091032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.100377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.064504] Lustre: server umount lustre-MDT0000 complete [ 2826.211722] LustreError: 93995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.234467] LustreError: 93995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2827.062501] LustreError: 97264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142415 with bad export cookie 1706510020629109235 [ 2827.069086] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2827.081706] LustreError: 97264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2827.512108] Lustre: server umount lustre-MDT0001 complete [ 2841.442967] Lustre: server umount lustre-OST0000 complete [ 2852.889085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.311390] LustreError: 98295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.459349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.906095] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2858.466400] LustreError: 98296:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.590229] LustreError: 98295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.398642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.217703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.075361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2881.229850] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2886.715854] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2900.749251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.012548] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2901.019799] LustreError: 98284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2901.046229] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2901.061686] LustreError: 100393:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2901.066511] LustreError: 100393:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2901.075219] LustreError: 100393:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2901.139282] Lustre: server umount lustre-OST0000 complete [ 2901.143056] LustreError: 100393:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2910.308247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.489707] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2910.503664] LustreError: 98284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2910.519731] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2910.531678] LustreError: 100860:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2910.539880] LustreError: 100860:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2910.542885] LustreError: 100860:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2910.617820] Lustre: server umount lustre-OST0000 complete [ 2910.632793] LustreError: 100860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2920.728821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2920.986682] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2920.994614] Lustre: Found index 0 for lustre-OST0000, updating log [ 2921.000692] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2921.112226] Lustre: lustre-OST0000: new disk, initializing [ 2921.116824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2921.127937] Lustre: 101394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2921.479468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.938615] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2932.988064] Lustre: Failing over lustre-OST0000 [ 2933.123357] Lustre: server umount lustre-OST0000 complete [ 2939.361413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2939.375938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.402580] Lustre: Skipped 1 previous similar message [ 2939.408050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2939.415182] Lustre: Skipped 2 previous similar messages [ 2942.434308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.436592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.554168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.562258] Lustre: Skipped 2 previous similar messages [ 2948.652959] Lustre: server umount lustre-MDT0000 complete [ 2952.651690] LustreError: 98275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142540 with bad export cookie 1706510020629111979 [ 2952.663406] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2952.668641] LustreError: 98275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2952.975742] Lustre: server umount lustre-MDT0001 complete [ 2961.379650] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:02:27 (1789142547) [ 2978.426316] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2988.375931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.862241] LustreError: 103741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.004402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.886597] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3002.704602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3008.838341] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3012.023680] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3017.717607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3018.347575] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3024.203703] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3027.560858] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3027.565561] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3027.578930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3031.880886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.024565] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3032.116338] Lustre: lustre-OST0001: new disk, initializing [ 3032.120654] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3032.129746] Lustre: 106060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3032.381920] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 3032.405703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3032.413100] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3032.419724] Lustre: Skipped 2 previous similar messages [ 3033.448189] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 3033.473792] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 3033.474576] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3033.483424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3038.310245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3038.336492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3038.490787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3038.505656] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3046.218503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3052.103653] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3056.952583] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 3059.182895] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.198074] Lustre: Skipped 1 previous similar message [ 3062.837455] Lustre: server umount lustre-MDT0001 complete [ 3064.296040] LustreError: 104949:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.311760] LustreError: 104949:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3072.336762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.412556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3072.415964] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3072.456941] Lustre: 103730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3073.724637] LustreError: 107602:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3073.737405] LustreError: 107602:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3073.757258] LustreError: 103733:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3073.870197] LustreError: 107613:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 3073.875563] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3073.875563] [ 3077.315818] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3078.151979] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3087.444713] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 12:04:34 (1789142674) [ 3104.758785] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3124.087875] Lustre: 109673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3145.055630] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3148.250721] Lustre: 110807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3236.834671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.840065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.858907] Lustre: Skipped 3 previous similar messages [ 3236.864218] Lustre: Skipped 6 previous similar messages [ 3243.008711] Lustre: server umount lustre-MDT0000 complete [ 3246.883068] LustreError: 104487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142834 with bad export cookie 1706510020629112980 [ 3246.893855] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3246.899215] LustreError: 104487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3247.082125] LustreError: 103736:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.100127] LustreError: 103736:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 3247.343547] Lustre: server umount lustre-MDT0001 complete [ 3253.902734] Lustre: server umount lustre-OST0000 complete [ 3257.943586] Lustre: server umount lustre-OST0001 complete [ 3263.357363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3273.788679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3282.249602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3292.938886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3313.301255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3325.041551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.075843] Lustre: lustre-MDT0000: reset Object Index mappings [ 3325.532567] LustreError: 114753:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.629768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.641268] Lustre: Skipped 1 previous similar message [ 3330.051684] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3338.231544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.258539] Lustre: lustre-MDT0001: reset Object Index mappings [ 3343.430623] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3346.494265] Lustre: 115900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3352.996398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.179365] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3364.054148] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3368.237862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.966217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3370.999536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3371.159786] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3375.970595] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3383.989818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3387.911222] Lustre: 117779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3442.351751] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 12:10:29 (1789143029) [ 3471.849525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3471.854624] LustreError: Skipped 1 previous similar message [ 3471.859783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.873571] Lustre: Skipped 2 previous similar messages [ 3471.886125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.899328] Lustre: Skipped 7 previous similar messages [ 3474.947483] Lustre: server umount lustre-MDT0000 complete [ 3478.500391] LustreError: 117189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.518842] LustreError: 117189:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 3478.563390] LustreError: 114735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143066 with bad export cookie 1706510020629160118 [ 3478.574786] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.584716] LustreError: 114735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3485.010710] Lustre: server umount lustre-MDT0001 complete [ 3495.497715] Lustre: server umount lustre-OST0000 complete [ 3505.422564] Lustre: server umount lustre-OST0001 complete [ 3511.396478] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3520.540154] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3558.331973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.540834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3558.569325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3558.633972] Lustre: lustre-MDT0000: new disk, initializing [ 3558.710418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.718549] Lustre: Skipped 3 previous similar messages [ 3558.739634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3563.432598] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3570.542634] Lustre: server umount lustre-MDT0000 complete [ 3579.994419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.317899] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3580.336225] Lustre: 122046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3580.357260] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3584.977347] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3589.533769] Lustre: server umount lustre-MDT0000 complete [ 3598.173641] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 12:13:04 (1789143184) [ 3599.955569] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3602.142355] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 12:13:08 (1789143188) [ 3620.799180] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3630.222237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.665927] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3642.072102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.175710] Lustre: 124095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3642.225658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3642.231161] Lustre: Skipped 1 previous similar message [ 3642.289837] Lustre: lustre-MDT0001: new disk, initializing [ 3642.401036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.410462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3646.670975] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3651.301761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3657.515991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.745547] Lustre: lustre-OST0000: new disk, initializing [ 3657.748751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3657.753735] Lustre: 125673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3659.784647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3659.791066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3659.921209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3663.228580] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3672.337239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3672.490544] Lustre: lustre-OST0001: new disk, initializing [ 3672.495381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3672.506558] Lustre: 126543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3674.275092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3674.285447] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3674.357324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3678.360232] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3687.822834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3691.270297] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4101.552922] Lustre: 132961:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4101.560449] Lustre: 132961:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4101.565788] Lustre: 132961:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4101.573919] Lustre: 132961:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4101.579813] Lustre: 132961:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4101.585423] Lustre: 132961:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4420.368326] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:26:46 (1789144006) [ 4435.426810] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.430634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.441757] Lustre: Skipped 8 previous similar messages [ 4435.449302] Lustre: Skipped 8 previous similar messages [ 4441.765520] Lustre: server umount lustre-MDT0000 complete [ 4445.676391] LustreError: 125807:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.687311] LustreError: 125807:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4445.716485] LustreError: 124088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144033 with bad export cookie 1706510020629229348 [ 4445.727234] LustreError: 124088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4445.727599] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.787540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.802966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4450.809087] Lustre: Skipped 1 previous similar message [ 4450.815799] Lustre: Skipped 5 previous similar messages [ 4452.568211] Lustre: server umount lustre-MDT0001 complete [ 4462.756024] Lustre: server umount lustre-OST0000 complete [ 4472.739981] Lustre: server umount lustre-OST0001 complete [ 4480.235955] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 4491.129252] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4528.892686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.156979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4529.225082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4529.346526] Lustre: lustre-MDT0000: new disk, initializing [ 4529.483030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.489030] Lustre: Skipped 5 previous similar messages [ 4529.519075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4534.521230] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4545.683674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.860622] Lustre: 136022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4545.935646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4545.941480] Lustre: Skipped 1 previous similar message [ 4546.059362] Lustre: lustre-MDT0001: new disk, initializing [ 4546.222200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4546.230770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4551.236790] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4562.081703] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.073343] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4576.227900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4576.521208] Lustre: lustre-OST0000: new disk, initializing [ 4576.525563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4576.539448] Lustre: 137990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4576.652743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.667203] Lustre: Skipped 1 previous similar message [ 4578.540331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4578.557228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4578.693906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4584.055303] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4594.583656] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4606.352829] Lustre: lustre-MDD0000: changelog on [ 4609.201428] Lustre: lustre-MDD0001: changelog on [ 6859.297943] Lustre: lustre-MDD0001: changelog off [ 6861.737924] Lustre: 136031:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6862.861730] Lustre: lustre-MDD0000: changelog off [ 6872.546057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6872.558896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.593382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6872.602344] Lustre: Skipped 2 previous similar messages [ 6876.938734] Lustre: server umount lustre-OST0000 complete [ 6882.785772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.793716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6882.804469] Lustre: Skipped 2 previous similar messages [ 6882.827375] Lustre: Skipped 1 previous similar message [ 6887.863927] Lustre: server umount lustre-MDT0000 complete [ 6887.906902] LustreError: 136905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.926182] LustreError: 136905:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6891.986416] LustreError: 136016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146479 with bad export cookie 1706510020630052702 [ 6891.996689] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6892.004280] LustreError: 136016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6892.459490] Lustre: server umount lustre-MDT0001 complete [ 6905.695207] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6908.274415] Key type lgssc unregistered [ 6908.517809] LNet: 147203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6908.523982] LNetError: 147203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6908.546139] LNet: Removed LNI 192.168.204.114@tcp [ 6909.433150] Key type .llcrypt unregistered [ 6909.435410] Key type ._llcrypt unregistered [ 6933.092104] Key type ._llcrypt registered [ 6933.093779] Key type .llcrypt registered [ 6933.208576] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6950.512945] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6951.724376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6951.831555] alg: No test for adler32 (adler32-zlib) [ 6952.928317] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6953.278676] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6955.057700] Key type lgssc registered [ 6956.428914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6994.439773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6994.470482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.718655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.751645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6995.815833] Lustre: lustre-MDT0000: new disk, initializing [ 6995.892335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.914304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7000.282891] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7010.656139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.768343] Lustre: 150412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7010.814323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7010.821261] Lustre: Skipped 1 previous similar message [ 7010.881352] Lustre: lustre-MDT0001: new disk, initializing [ 7010.947198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7010.975461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7010.986444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7015.815133] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7025.872131] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7033.025657] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7040.522855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.829162] Lustre: lustre-OST0000: new disk, initializing [ 7040.835466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7040.843437] Lustre: 152378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7040.932670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.993234] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7049.796161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7049.806080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7049.910381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7059.768733] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7066.695327] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_module obdecho/obdecho [ 7068.795064] LustreError: 153382:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7068.805993] LustreError: 153382:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7070.891616] LustreError: 153479:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7086.052026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.059081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7086.076632] Lustre: Skipped 2 previous similar messages [ 7086.079552] Lustre: Skipped 1 previous similar message [ 7090.274225] Lustre: server umount lustre-MDT0000 complete [ 7094.769676] LustreError: 150403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146682 with bad export cookie 6548226055900752814 [ 7094.777754] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.800231] LustreError: 150403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7095.878628] Lustre: server umount lustre-MDT0001 complete [ 7111.946955] Lustre: server umount lustre-OST0000 complete [ 7134.869929] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7151.638977] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7195.757804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.968795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7195.999718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7196.088954] Lustre: lustre-MDT0000: new disk, initializing [ 7196.153953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.172200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7200.525759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7212.250734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7212.394885] Lustre: 157220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7212.439627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7212.443218] Lustre: Skipped 1 previous similar message [ 7212.501382] Lustre: lustre-MDT0001: new disk, initializing [ 7212.594379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.657620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7212.670974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7217.864986] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7228.323542] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7234.771233] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7243.172473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.451852] Lustre: lustre-OST0000: new disk, initializing [ 7243.456302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7243.465334] Lustre: 159188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7243.576668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.336719] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7252.042305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7252.052755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7252.256781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7262.114404] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7267.860311] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7268.241899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7273.550996] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7273.820883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7278.053985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7278.055584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.075295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7278.093911] Lustre: Skipped 1 previous similar message [ 7278.107335] Lustre: Skipped 2 previous similar messages [ 7283.172243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7283.187208] Lustre: Skipped 1 previous similar message [ 7288.305797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7288.321676] Lustre: Skipped 1 previous similar message [ 7290.336232] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7290.553592] Lustre: server umount lustre-OST0000 complete [ 7298.539241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.545622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.557963] Lustre: Skipped 1 previous similar message [ 7298.591105] Lustre: Skipped 1 previous similar message [ 7302.239495] Lustre: server umount lustre-MDT0000 complete [ 7303.650807] LustreError: 157231:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.684218] LustreError: 157231:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7307.909838] LustreError: 158014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146895 with bad export cookie 6548226055900753773 [ 7307.918206] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7307.927236] LustreError: 158014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7308.477540] Lustre: server umount lustre-MDT0001 complete [ 7313.090896] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:14:59 (1789146899) [ 7314.959435] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7317.072584] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 13:15:03 (1789146903) [ 7326.206457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7326.764833] LustreError: 161606:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.885324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7332.306680] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7332.330962] LustreError: 161607:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.419754] LustreError: 161606:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.541841] LustreError: 161607:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7342.254954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.096515] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7355.888471] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7361.876890] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7368.961282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7369.288277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.293483] Lustre: Skipped 1 previous similar message [ 7376.113467] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7385.192836] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7398.881744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7398.895930] LustreError: Skipped 1 previous similar message [ 7398.911749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.928291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.989914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.805455] Lustre: server umount lustre-MDT0000 complete [ 7406.350183] LustreError: 161602:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.087028] LustreError: 161587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146997 with bad export cookie 6548226055900754655 [ 7409.092826] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.096526] LustreError: 161587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7413.220000] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.240090] Lustre: Skipped 1 previous similar message [ 7415.610445] Lustre: server umount lustre-MDT0001 complete [ 7423.464826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7438.471910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.734362] LustreError: 165463:0:(llog.c:1655:llog_backup()) MGC192.168.204.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7438.744567] Lustre: 165463:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7439.844342] LustreError: 165468:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.868097] LustreError: 165468:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 7471.868849] LustreError: 165470:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.890705] LustreError: 165470:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 25 previous similar messages [ 7499.744381] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7499.755995] Lustre: 165463:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7536.098292] LustreError: 165469:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7536.125142] LustreError: 165469:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 40 previous similar messages [ 7540.748240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7545.851847] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7546.413706] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7556.829765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.416545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7586.299954] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7587.021456] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7599.288987] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7601.704136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7609.330094] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7611.462874] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7632.354761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.365114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7632.366499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7632.366504] Lustre: Skipped 7 previous similar messages [ 7632.378288] Lustre: Skipped 1 previous similar message [ 7633.769317] Lustre: server umount lustre-MDT0000 complete [ 7637.645292] LustreError: 161587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147225 with bad export cookie 6548226055900761144 [ 7637.652207] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7637.660516] LustreError: 161587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7637.973398] Lustre: server umount lustre-MDT0001 complete [ 7653.578971] Lustre: server umount lustre-OST0000 complete [ 7678.182255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7692.688169] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7733.526908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.795604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7733.841771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7733.951716] Lustre: lustre-MDT0000: new disk, initializing [ 7734.049526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7734.060449] Lustre: Skipped 1 previous similar message [ 7734.073955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7739.302488] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7753.666761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.894660] Lustre: 170576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7754.314316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7761.504099] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7772.825527] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7779.605870] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7787.819195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.165742] Lustre: lustre-OST0000: new disk, initializing [ 7788.169253] Lustre: Skipped 1 previous similar message [ 7788.173810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7788.178610] Lustre: Skipped 2 previous similar messages [ 7788.184170] Lustre: 172544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7795.352775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7795.379401] Lustre: Skipped 1 previous similar message [ 7795.387300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7795.641282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7796.388781] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7807.519186] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7813.176071] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7813.380679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7819.244673] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7819.520749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7826.405038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7826.424187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.449839] Lustre: Skipped 1 previous similar message [ 7826.468406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.472578] Lustre: Skipped 3 previous similar messages [ 7828.116696] Lustre: server umount lustre-OST0000 complete [ 7836.139730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7836.160529] LustreError: Skipped 1 previous similar message [ 7839.694799] Lustre: server umount lustre-MDT0000 complete [ 7841.763873] LustreError: 170582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.800975] LustreError: 170582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 40 previous similar messages [ 7844.500411] LustreError: 170568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147432 with bad export cookie 6548226055900777951 [ 7844.515792] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7844.519246] LustreError: 170568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7845.133465] Lustre: server umount lustre-MDT0001 complete [ 7855.116097] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:24:01 (1789147441) [ 7862.651503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.537852] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7877.652877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7883.497805] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7892.471845] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7898.485063] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7905.928340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7913.528389] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7922.117278] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7936.045087] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:25:22 (1789147522) [ 7943.144621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.146273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.151540] Lustre: Skipped 3 previous similar messages [ 7943.153269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7943.153276] Lustre: Skipped 2 previous similar messages [ 7947.265702] Lustre: server umount lustre-OST0000 complete [ 7957.945787] Lustre: server umount lustre-MDT0000 complete [ 7962.619119] LustreError: 174937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147550 with bad export cookie 6548226055900778833 [ 7962.625466] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7962.640033] LustreError: 174937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7963.042450] Lustre: server umount lustre-MDT0001 complete [ 7974.651745] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7977.411480] Key type lgssc unregistered [ 7977.740197] LNet: 178693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7977.749513] LNetError: 178693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7977.764994] LNet: Removed LNI 192.168.204.114@tcp [ 7978.746830] Key type .llcrypt unregistered [ 7978.749772] Key type ._llcrypt unregistered [ 7999.958107] Key type ._llcrypt registered [ 7999.962455] Key type .llcrypt registered [ 8000.081068] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8001.205619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8001.419173] alg: No test for adler32 (adler32-zlib) [ 8002.592683] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8002.787692] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8004.448980] Key type lgssc registered [ 8006.145736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8015.034474] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8015.048908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.561150] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8035.681767] LustreError: 179936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 8040.802918] LustreError: 179936:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8040.820188] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8040.827393] LustreError: MGC192.168.204.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8040.841968] LustreError: 179936:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8040.851666] LustreError: 179936:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8040.856703] LustreError: 179936:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8040.922655] Lustre: server umount lustre-OST0000 complete [ 8040.926734] LustreError: 179936:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8051.317541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.845934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.875246] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8062.442240] LustreError: 180436:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.466659] LustreError: 180436:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8067.562266] LustreError: 180437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.200657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.558489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8073.819857] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8083.067528] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8089.301527] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8097.202469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8097.465555] LustreError: 182257:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8097.476256] Lustre: 182257:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8097.686207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.595990] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8115.774534] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8119.785326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8119.797552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.815180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8123.367606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8123.375053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8123.417548] Lustre: Skipped 1 previous similar message [ 8125.633836] Lustre: server umount lustre-MDT0000 complete [ 8128.495496] LustreError: 180433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8128.521163] LustreError: 180433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8129.829976] LustreError: 180418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147717 with bad export cookie 16322673403237979115 [ 8129.838593] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8129.853060] LustreError: 180418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8130.502841] Lustre: server umount lustre-MDT0001 complete [ 8145.951612] Lustre: server umount lustre-OST0000 complete [ 8158.873312] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8171.163157] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8211.431152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.732709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8211.767707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8211.843957] Lustre: lustre-MDT0000: new disk, initializing [ 8211.918931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8211.942234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8216.792461] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8228.478820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8228.630348] Lustre: 185784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8228.693180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8228.697578] Lustre: Skipped 1 previous similar message [ 8228.788324] Lustre: lustre-MDT0001: new disk, initializing [ 8228.940472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8229.016535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8229.044202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8234.207799] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8244.183730] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8249.693231] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8257.316667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8257.667184] Lustre: lustre-OST0000: new disk, initializing [ 8257.670873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8257.687152] Lustre: 187751:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8257.787572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.295179] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8266.301327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8266.316037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8266.420630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8274.362493] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8279.530921] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8279.816920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8286.590148] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8286.899637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8292.325119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8292.331573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.335112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8292.335125] Lustre: Skipped 2 previous similar messages [ 8292.386209] Lustre: Skipped 1 previous similar message [ 8295.698295] Lustre: server umount lustre-OST0000 complete [ 8300.514636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8300.519953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.549578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.565804] Lustre: Skipped 1 previous similar message [ 8306.163341] Lustre: server umount lustre-MDT0000 complete [ 8307.687192] LustreError: 185793:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.724254] LustreError: 185793:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8310.325680] LustreError: 185776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147898 with bad export cookie 16322673403237979836 [ 8310.330921] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8310.336881] LustreError: 185776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8310.731449] Lustre: server umount lustre-MDT0001 complete [ 8321.018293] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:31:47 (1789147907) [ 8341.246171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8344.381808] Key type lgssc unregistered [ 8344.720711] LNet: 190700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8344.742953] LNetError: 190700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8344.761237] LNet: Removed LNI 192.168.204.114@tcp [ 8345.958671] Key type .llcrypt unregistered [ 8345.962051] Key type ._llcrypt unregistered [ 8367.734189] Key type ._llcrypt registered [ 8367.736318] Key type .llcrypt registered [ 8367.847576] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8368.985872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8369.009812] alg: No test for adler32 (adler32-zlib) [ 8370.258664] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8370.535714] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8372.232330] Key type lgssc registered [ 8373.514439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8381.972362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8381.989278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8383.428311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8389.209062] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8393.699717] LustreError: 191922:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.734243] LustreError: 191922:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8398.832694] LustreError: 191921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8399.144501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8399.546703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8404.227862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8415.026155] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.323491] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8433.103409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.332200] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8433.335169] LustreError: 193787:0:(llog.c:1696:llog_backup()) MGC192.168.204.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8433.342216] Lustre: 193787:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8433.471272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.022487] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8448.283933] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8451.042799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8451.054049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.070496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.140335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.147301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.159813] Lustre: Skipped 1 previous similar message [ 8455.187936] Lustre: Skipped 1 previous similar message [ 8456.897863] Lustre: server umount lustre-MDT0000 complete [ 8460.258979] LustreError: 191921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.289672] LustreError: 191921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8460.768466] LustreError: 191901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148048 with bad export cookie 16549173959693761126 [ 8460.778565] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8460.786095] LustreError: 191901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8461.122153] Lustre: server umount lustre-MDT0001 complete [ 8475.907125] Lustre: server umount lustre-OST0000 complete [ 8486.838893] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8500.299532] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8543.736739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8544.043582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8544.132749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8544.235543] Lustre: lustre-MDT0000: new disk, initializing [ 8544.375282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.417924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8550.246494] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8562.195871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.343661] Lustre: 197316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8562.460622] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8562.468936] Lustre: Skipped 1 previous similar message [ 8562.481490] LustreError: 198100:0:(llog.c:1696:llog_backup()) MGC192.168.204.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8562.498172] LustreError: 198100:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8562.506129] Lustre: 198100:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8562.573861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8562.577072] Lustre: Skipped 1 previous similar message [ 8562.700267] Lustre: lustre-MDT0001: new disk, initializing [ 8562.724488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8562.732225] Lustre: Skipped 1 previous similar message [ 8562.833736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8562.881066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8562.896065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8567.436920] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8577.078313] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8583.834284] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8591.976583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8592.401795] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8592.403857] Lustre: Skipped 1 previous similar message [ 8592.412167] LustreError: 199262:0:(llog.c:1696:llog_backup()) MGC192.168.204.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8592.421262] LustreError: 199262:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8592.428830] Lustre: 199262:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8592.437721] Lustre: 199262:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8592.457983] Lustre: lustre-OST0000: new disk, initializing [ 8592.463249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8592.469207] Lustre: 199286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8592.567661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8598.137158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8598.148383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8598.277984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8599.971672] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8610.649522] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8616.570676] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8616.919115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8623.853775] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8624.328234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8629.231584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.232239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.254698] LustreError: Skipped 1 previous similar message [ 8629.261791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.352473] Lustre: Skipped 1 previous similar message [ 8633.397283] Lustre: server umount lustre-OST0000 complete [ 8639.460606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.464093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.479165] Lustre: Skipped 1 previous similar message [ 8639.500362] Lustre: Skipped 2 previous similar messages [ 8643.682941] Lustre: server umount lustre-MDT0000 complete [ 8644.589537] LustreError: 197328:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8644.616427] LustreError: 197328:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8647.404729] LustreError: 197309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148235 with bad export cookie 16549173959693761840 [ 8647.408658] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.414163] LustreError: 197309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8647.793158] Lustre: server umount lustre-MDT0001 complete [ 8656.648768] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:37:23 (1789148243) [ 8676.348862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8686.956350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.532253] LustreError: 202416:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8687.576634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8692.679776] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8697.828273] LustreError: 202417:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.851278] LustreError: 202417:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8702.659023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.596548] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8710.915499] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8718.367116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.064281] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8734.224859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8734.398528] Lustre: lustre-OST0001: new disk, initializing [ 8734.411643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8734.420180] Lustre: 204724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8734.535881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8734.547746] Lustre: Skipped 2 previous similar messages [ 8740.508634] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8740.947213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8740.959582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8741.122351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8750.988410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8759.981867] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8762.364863] Lustre: DEBUG MARKER: ADD OST3 [ 8767.806206] loop: module loaded [ 8767.881333] loop0: detected capacity change from 0 to 204800000 [ 8768.129965] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8768.156247] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8768.175880] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8774.308107] loop0: detected capacity change from 0 to 204800000 [ 8775.105554] Lustre: DEBUG MARKER: STOP OST3 [ 8780.081009] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8780.297565] Lustre: lustre-OST0003: new disk, initializing [ 8780.304406] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8780.310292] Lustre: 206722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8781.836805] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8781.990409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8782.003186] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8782.078763] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8783.905075] LustreError: 206880:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8784.005537] Lustre: Failing over lustre-OST0003 [ 8784.153499] Lustre: server umount lustre-OST0003 complete [ 8786.959525] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8787.428640] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8787.436895] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.446678] LustreError: 203902:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.451842] Lustre: Skipped 1 previous similar message [ 8795.643381] Lustre: DEBUG MARKER: START OST3 again [ 8802.503712] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.807666] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8802.818531] Lustre: Skipped 1 previous similar message [ 8802.849053] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8803.075368] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8804.316106] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8804.316909] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8810.354647] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8833.113550] Lustre: Failing over lustre-OST0003 [ 8833.256442] Lustre: server umount lustre-OST0003 complete [ 8833.528968] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.535031] LustreError: 203901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8833.541528] Lustre: Skipped 1 previous similar message [ 8833.555158] LustreError: 203901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 8853.986115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.991282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8854.010324] Lustre: Skipped 2 previous similar messages [ 8854.039883] Lustre: Skipped 3 previous similar messages [ 8859.381625] Lustre: server umount lustre-MDT0000 complete [ 8863.866135] LustreError: 204723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148451 with bad export cookie 16549173959693762722 [ 8863.881343] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.900412] LustreError: 204723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8870.287312] Lustre: server umount lustre-MDT0001 complete [ 8874.953745] Lustre: server umount lustre-OST0000 complete [ 8879.820759] Lustre: server umount lustre-OST0001 complete [ 8888.627767] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8900.196871] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8942.973247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.272305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8943.312844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8943.419847] Lustre: lustre-MDT0000: new disk, initializing [ 8943.502685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8943.519145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8948.816427] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8960.584477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.738565] Lustre: 212184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8960.796970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8960.802769] Lustre: Skipped 1 previous similar message [ 8960.877317] Lustre: lustre-MDT0001: new disk, initializing [ 8960.999268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8961.021302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8966.167534] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8977.308892] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8982.938375] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8991.316316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8991.856133] Lustre: 214155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8993.535518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8993.572386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8993.907569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9000.209175] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9011.128144] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9017.095704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9017.347894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9022.955466] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9023.174653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9029.605657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9029.607939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9029.620687] LustreError: Skipped 1 previous similar message [ 9029.625176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9029.643117] Lustre: Skipped 4 previous similar messages [ 9029.669046] Lustre: Skipped 9 previous similar messages [ 9039.328941] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9039.569042] Lustre: server umount lustre-OST0000 complete [ 9050.637352] Lustre: server umount lustre-MDT0000 complete [ 9055.155562] LustreError: 212179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148643 with bad export cookie 16549173959693765011 [ 9055.163380] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.181085] LustreError: 212179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9055.598456] Lustre: server umount lustre-MDT0001 complete [ 9059.554252] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:44:06 (1789148646) [ 9076.435089] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9087.527408] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9129.563457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.023238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9130.064409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9130.070981] Lustre: Skipped 1 previous similar message [ 9130.226154] Lustre: lustre-MDT0000: new disk, initializing [ 9130.231501] Lustre: Skipped 1 previous similar message [ 9130.398308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9135.129218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9146.166641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.303386] Lustre: 218770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9146.618540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9151.190387] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9161.255392] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9168.341401] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9177.576489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9177.992340] Lustre: 220740:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9179.474214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9179.583419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9184.453645] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9195.071614] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9201.671521] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9201.965587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9207.219970] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9207.508810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9210.338927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9210.339924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.350306] LustreError: Skipped 1 previous similar message [ 9210.355686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.379297] Lustre: Skipped 3 previous similar messages [ 9210.410793] Lustre: Skipped 7 previous similar messages [ 9216.545984] Lustre: server umount lustre-OST0000 complete [ 9223.650782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9227.907629] Lustre: server umount lustre-MDT0000 complete [ 9230.823888] LustreError: 218777:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.849152] LustreError: 218777:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 9232.813785] LustreError: 218763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148820 with bad export cookie 16549173959693765900 [ 9232.824486] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9232.832173] LustreError: 218763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9233.417493] Lustre: server umount lustre-MDT0001 complete [ 9243.581703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.048210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.055865] Lustre: Skipped 5 previous similar messages [ 9248.852782] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9258.043432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.659918] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9271.530694] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9277.729863] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9284.569647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9292.450208] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9301.434906] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9313.770125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9313.771323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9313.777469] LustreError: Skipped 1 previous similar message [ 9313.783959] Lustre: Skipped 5 previous similar messages [ 9317.518575] Lustre: server umount lustre-OST0000 complete [ 9328.594692] Lustre: server umount lustre-MDT0000 complete [ 9332.609474] LustreError: 222942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148920 with bad export cookie 16549173959693766782 [ 9332.613173] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9332.627614] LustreError: 222942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9333.280315] Lustre: server umount lustre-MDT0001 complete [ 9444.819628] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9449.097513] Key type lgssc unregistered [ 9449.359402] LNet: 226630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9449.372593] LNetError: 226630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9450.422629] LNet: Removed LNI 192.168.204.114@tcp [ 9451.926220] Key type .llcrypt unregistered [ 9451.928621] Key type ._llcrypt unregistered [ 9485.158288] Key type ._llcrypt registered [ 9485.162228] Key type .llcrypt registered [ 9485.353256] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9504.890988] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9506.578702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9506.628619] alg: No test for adler32 (adler32-zlib) [ 9507.785244] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 9508.123658] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9509.824236] Key type lgssc registered [ 9511.103451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9555.614628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9555.649935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.992069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9557.034126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9557.174150] Lustre: lustre-MDT0000: new disk, initializing [ 9557.355240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.379340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9562.639548] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9575.907714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.022668] Lustre: 229985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9576.077437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9576.081799] Lustre: Skipped 1 previous similar message [ 9576.167585] Lustre: lustre-MDT0001: new disk, initializing [ 9576.318573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9576.362696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9576.371950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9581.549225] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9591.900912] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9598.431889] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9606.315883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9606.690820] Lustre: lustre-OST0000: new disk, initializing [ 9606.699505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9606.708053] Lustre: 231953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9606.851543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.960095] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9616.568588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9616.587490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9616.850615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9629.022547] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9634.822404] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9635.252569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9641.630073] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9642.009289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9647.592532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.593710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9647.628940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9647.641546] Lustre: Skipped 1 previous similar message [ 9650.633065] Lustre: server umount lustre-OST0000 complete [ 9657.827385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.831096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.845694] Lustre: Skipped 2 previous similar messages [ 9657.858255] Lustre: Skipped 1 previous similar message [ 9662.439660] Lustre: server umount lustre-MDT0000 complete [ 9666.427306] LustreError: 229978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149254 with bad export cookie 9922207056608573393 [ 9666.441797] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.446790] LustreError: 229978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9666.929882] Lustre: server umount lustre-MDT0001 complete [ 9695.221513] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9706.363665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.817304] LustreError: 234927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.831831] LustreError: 234927:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9706.895262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.490806] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9712.105174] LustreError: 234928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.196744] LustreError: 234927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.315566] LustreError: 234928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.533450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.786136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9726.656315] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9729.779913] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9737.812651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.288797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.603495] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9754.128970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9754.296685] Lustre: lustre-OST0001: new disk, initializing [ 9754.300944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9754.307887] Lustre: 237236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9754.439250] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9760.839996] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9763.473105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9763.484975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9763.579933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9774.453637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9783.193621] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9799.201470] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:56:25 (1789149385) [ 9803.745157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9803.753512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.770604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9809.906880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9809.916928] Lustre: Skipped 6 previous similar messages [ 9815.015808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9815.033431] Lustre: Skipped 3 previous similar messages [ 9816.033137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9816.292882] Lustre: server umount lustre-MDT0000 complete [ 9820.137993] LustreError: 234928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.154888] LustreError: 234928:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9820.643826] LustreError: 234909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149408 with bad export cookie 9922207056608574282 [ 9820.657023] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9820.663764] LustreError: 234909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9821.238223] Lustre: server umount lustre-MDT0001 complete [ 9836.146364] Lustre: server umount lustre-OST0000 complete [ 9850.723859] Lustre: server umount lustre-OST0001 complete [ 9858.320429] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9873.109571] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9920.987267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9921.303722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9921.333805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9921.465326] Lustre: lustre-MDT0000: new disk, initializing [ 9921.621074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9921.663175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9926.859965] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9937.875122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9937.989902] Lustre: 241197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9938.053942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9938.060416] Lustre: Skipped 1 previous similar message [ 9938.170892] Lustre: lustre-MDT0001: new disk, initializing [ 9938.345886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9938.356706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9944.749146] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9956.150519] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9962.222051] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9970.026290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9970.313921] Lustre: lustre-OST0000: new disk, initializing [ 9970.320251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9970.328861] Lustre: 243167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9971.661406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9971.675298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9971.822823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9977.641506] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9989.469881] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.472552] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9995.856726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10000.523915] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10000.710597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10007.535269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10007.538248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.550460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10007.566445] Lustre: Skipped 4 previous similar messages [10009.001259] Lustre: server umount lustre-OST0000 complete [10015.202475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10015.208828] LustreError: Skipped 1 previous similar message [10015.212416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.640909] Lustre: server umount lustre-MDT0000 complete [10022.881875] LustreError: 241209:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.894639] LustreError: 241209:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10024.076698] LustreError: 241189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149612 with bad export cookie 9922207056608576417 [10024.081858] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10024.093345] LustreError: 241189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10024.438966] Lustre: server umount lustre-MDT0001 complete [10033.444820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.959688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.968240] Lustre: Skipped 2 previous similar messages [10039.271885] LustreError: 245389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10039.291279] LustreError: 245389:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10039.457823] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10048.006136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.730136] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10062.437272] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10070.256671] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10077.922637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10085.046751] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10092.979260] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10242.022526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10242.025038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.065757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10242.084748] Lustre: Skipped 2 previous similar messages [10242.116228] Lustre: Skipped 5 previous similar messages [10253.280164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10253.452294] Lustre: server umount lustre-OST0000 complete [10258.403341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10258.419814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.462275] Lustre: server umount lustre-MDT0000 complete [10267.628384] LustreError: 245389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.642613] LustreError: 245389:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10268.993351] LustreError: 245370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149856 with bad export cookie 9922207056608577299 [10269.010991] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10269.524467] Lustre: server umount lustre-MDT0001 complete [10362.262659] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [10364.950910] Key type lgssc unregistered [10365.216333] LNet: 249019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10365.225129] LNetError: 249019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10365.248079] LNet: Removed LNI 192.168.204.114@tcp [10366.201323] Key type .llcrypt unregistered [10366.204955] Key type ._llcrypt unregistered [10391.042237] Key type ._llcrypt registered [10391.044630] Key type .llcrypt registered [10391.121552] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10407.253463] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10408.379990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10408.507062] alg: No test for adler32 (adler32-zlib) [10409.723774] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [10410.049957] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [10411.784384] Key type lgssc registered [10413.090986] Lustre: Echo OBD driver; http://www.lustre.org/ [10473.995531] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10484.940682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10484.957881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.260261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10486.290356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10486.361341] Lustre: lustre-MDT0000: new disk, initializing [10486.423961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10486.443648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10491.239223] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10503.702228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.809360] Lustre: 253114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10503.886378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10503.895378] Lustre: Skipped 1 previous similar message [10503.981336] Lustre: lustre-MDT0001: new disk, initializing [10504.125886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10504.197951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10504.206210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10508.766313] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10513.543670] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10521.498081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.845191] Lustre: lustre-OST0000: new disk, initializing [10521.852715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10521.867962] Lustre: 254744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10522.013973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10523.053547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10523.069797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10523.218726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10528.440284] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10540.738520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10540.979843] Lustre: lustre-OST0001: new disk, initializing [10540.991807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10541.008068] Lustre: 255617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10541.221418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10549.127526] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10550.438140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10550.444255] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10550.535425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10559.610098] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10568.532791] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10580.964429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10580.979960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.004504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.479404] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.496693] Lustre: Skipped 2 previous similar messages [10581.507100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.515705] Lustre: Skipped 3 previous similar messages [10581.996316] Lustre: server umount lustre-MDT0000 complete [10586.117764] LustreError: 253907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150174 with bad export cookie 14548883476555514399 [10586.128250] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.132267] LustreError: 253907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10586.607869] Lustre: server umount lustre-MDT0001 complete [10600.555025] Lustre: server umount lustre-OST0000 complete [10602.977131] Lustre: 250599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150174/real 1789150174] req@ffff8b7587244000 x1876059748380416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10603.005267] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.267515] Lustre: server umount lustre-OST0001 complete [10621.825622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10632.210583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10642.166843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10653.107888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10675.422363] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10689.487528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.511569] Lustre: lustre-MDT0000: reset Object Index mappings [10690.067796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10694.841768] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10695.139199] LustreError: 260625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.150785] LustreError: 260625:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10699.240575] LustreError: 260626:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.355557] LustreError: 260625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.754610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.776410] Lustre: lustre-MDT0001: reset Object Index mappings [10705.341839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10711.119156] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10715.215584] Lustre: 261769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10725.257273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10725.735602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10730.883917] LustreError: 262345:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.019829] LustreError: 262123:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.060064] LustreError: 262123:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10735.074706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10735.527386] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10745.352232] LustreError: 262122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.389520] LustreError: 262122:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10748.177827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.048859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10755.000420] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10762.268435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10766.475940] Lustre: 263641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10771.947888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10771.959701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.982795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10773.739217] Lustre: server umount lustre-MDT0000 complete [10774.505674] LustreError: 260626:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10774.529416] LustreError: 260626:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10778.168629] LustreError: 263644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150366 with bad export cookie 14548883476555521189 [10778.173472] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10778.178639] LustreError: 263644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10778.532860] Lustre: server umount lustre-MDT0001 complete [10793.096931] Lustre: server umount lustre-OST0000 complete [10796.005755] Lustre: 250599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150367/real 1789150367] req@ffff8b7588040b40 x1876059748433280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10796.039637] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10796.069089] Lustre: Skipped 3 previous similar messages [10797.160536] Lustre: server umount lustre-OST0001 complete [10817.839245] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:13:24 (1789150404) [10836.569210] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10850.097588] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10901.897322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.202878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10902.250733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10902.342665] Lustre: lustre-MDT0000: new disk, initializing [10902.435755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10902.441447] Lustre: Skipped 1 previous similar message [10902.454263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10906.643534] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10918.428900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.536983] Lustre: 267900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10918.591602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10918.594937] Lustre: Skipped 1 previous similar message [10918.650290] Lustre: lustre-MDT0001: new disk, initializing [10918.845127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10918.860650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10924.109165] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10934.903309] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10941.792611] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10949.809098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10950.263387] Lustre: lustre-OST0000: new disk, initializing [10950.272541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10950.286575] Lustre: 269867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10951.805748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10951.818038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10951.972239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10957.673272] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10968.517608] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10975.398572] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10975.770586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10982.334772] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10982.697948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10988.002373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.005632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10988.031067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10993.121674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10993.130272] Lustre: Skipped 2 previous similar messages [10999.265984] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10999.574542] Lustre: server umount lustre-OST0000 complete [11005.921819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11005.930173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11005.944453] Lustre: Skipped 1 previous similar message [11005.953777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11005.962237] Lustre: Skipped 2 previous similar messages [11010.663968] Lustre: server umount lustre-MDT0000 complete [11013.601277] LustreError: 267910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11013.616026] LustreError: 267910:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11014.212927] LustreError: 267892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150602 with bad export cookie 14548883476555523282 [11014.217243] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11014.242363] LustreError: 267892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11014.507684] Lustre: server umount lustre-MDT0001 complete [11035.198204] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11045.769519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11046.074617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11046.078311] Lustre: Skipped 2 previous similar messages [11050.564539] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11059.951982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11066.232754] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11069.590961] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11078.307082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11086.456858] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11098.822308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.045736] Lustre: lustre-OST0001: new disk, initializing [11099.050394] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11099.061787] Lustre: 275114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11100.572161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11100.580961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11100.696557] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11108.286741] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11119.839573] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11124.099248] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11130.238898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11130.279416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11141.617761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.622817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11141.632848] Lustre: Skipped 2 previous similar messages [11141.649077] Lustre: Skipped 5 previous similar messages [11144.696302] Lustre: server umount lustre-MDT0000 complete [11146.725525] LustreError: 273532:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11146.757540] LustreError: 273532:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [11148.609627] LustreError: 274314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150736 with bad export cookie 14548883476555524164 [11148.613961] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11148.626340] LustreError: 274314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11149.076306] Lustre: server umount lustre-MDT0001 complete [11164.017607] Lustre: server umount lustre-OST0000 complete [11167.712716] Lustre: 250599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150739/real 1789150739] req@ffff8b758a5621c0 x1876059748548608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11168.656442] Lustre: server umount lustre-OST0001 complete [11189.392686] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11203.235426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.978274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11203.986071] Lustre: Skipped 3 previous similar messages [11208.793099] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11217.362029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.672497] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11226.037322] Lustre: 279312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11232.813691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11240.691982] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11249.133448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11250.027741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11255.837234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11258.052792] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11265.763867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11270.461894] Lustre: 281191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11276.876454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11277.085356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11288.590883] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:21:15 (1789150875) [11304.935088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11304.951332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11304.971417] Lustre: Skipped 3 previous similar messages [11304.980065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11309.737495] Lustre: server umount lustre-MDT0000 complete [11312.117932] LustreError: 278172:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11312.138307] LustreError: 278172:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [11315.056590] LustreError: 279314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150903 with bad export cookie 14548883476555530471 [11315.064174] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11315.083744] LustreError: 279314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11321.659165] Lustre: server umount lustre-MDT0001 complete [11332.749182] Lustre: server umount lustre-OST0000 complete [11337.018851] Lustre: server umount lustre-OST0001 complete [11343.694495] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [11353.453540] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11402.975507] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11415.538461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.749809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11415.785977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11415.930385] Lustre: lustre-MDT0000: new disk, initializing [11416.022730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11421.154914] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11432.678742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11432.804637] Lustre: 285434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11433.099454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11437.905671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11449.820514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11450.194901] Lustre: 287067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11452.292621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11457.105605] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11468.409265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11468.614108] Lustre: 287935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11468.706722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11468.712504] Lustre: Skipped 6 previous similar messages [11469.769322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11469.779189] Lustre: Skipped 1 previous similar message [11469.868032] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11475.210453] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11485.981316] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11489.692969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11489.698825] Lustre: Skipped 1 previous similar message [11507.611491] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 14:24:54 (1789151094) [11510.758760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11510.771438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11510.781354] Lustre: Skipped 8 previous similar messages [11510.797847] Lustre: Skipped 9 previous similar messages [11512.440712] Lustre: server umount lustre-MDT0000 complete [11516.640432] LustreError: 285427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151104 with bad export cookie 14548883476555538332 [11516.647231] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11516.653367] LustreError: 285427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11517.238969] Lustre: server umount lustre-MDT0001 complete [11531.259679] Lustre: server umount lustre-OST0000 complete [11545.663656] Lustre: server umount lustre-OST0001 complete [11554.885373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11560.224980] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11569.318566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.822174] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11581.793984] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11588.002798] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11594.732854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11600.177084] LustreError: 292011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.209190] LustreError: 292011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [11601.657508] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11610.413105] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11631.756678] loop1: detected capacity change from 0 to 204800000 [11632.053110] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11632.063655] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11632.081963] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.125730] loop1: detected capacity change from 0 to 204800000 [11644.124358] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.237731] LustreError: 290173:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17929: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11644.248262] LustreError: 290173:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17929 [11644.252464] LustreError: 290173:0:(mgs_handler.c:517:mgs_target_reg()) fs17929-MDT0001: Error checking target index: rc = -13 [11644.261824] LustreError: fs17929-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11644.269936] LustreError: 293900:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11644.276407] LustreError: 293900:0:(tgt_mount.c:2129:server_put_super()) no obd fs17929-MDT0001 [11644.280692] LustreError: 293900:0:(tgt_mount.c:116:server_deregister_mount()) fs17929-MDT0001 not registered [11644.313103] Lustre: server umount fs17929-MDT0001 complete [11644.318401] LustreError: 293900:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11653.106278] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.220232] Lustre: Setting parameter fs17929-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17929-MDT0001 [11653.288042] Lustre: srv-fs17929-MDT0001: No data found on store. Initialize space. [11653.292493] Lustre: Skipped 4 previous similar messages [11653.439713] Lustre: fs17929-MDT0001: new disk, initializing [11653.443941] Lustre: Skipped 3 previous similar messages [11653.597339] Lustre: srv-fs17929-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11653.615394] Lustre: 294367:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17929-MDT0001-mdtlov: can't create new llog: rc = -115 [11653.634488] LustreError: 294367:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17929-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11658.697527] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11666.532512] Lustre: server umount fs17929-MDT0001 complete [11681.265846] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11684.293469] loop1: detected capacity change from 0 to 204800000 [11692.318838] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11693.362832] Lustre: 295938:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17929-MDT0001-mdtlov: can't create new llog: rc = -115 [11693.396790] LustreError: 295938:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17929-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11699.559987] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11705.635973] Lustre: server umount fs17929-MDT0001 complete [11718.113338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11718.131578] LustreError: Skipped 1 previous similar message [11718.141722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11724.213741] Lustre: server umount lustre-MDT0000 complete [11735.014609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.160925] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11740.634569] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11740.673928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11750.229331] loop1: detected capacity change from 0 to 204800000 [11756.167840] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11756.764581] Lustre: 298387:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17929-MDT0001-mdtlov: can't create new llog: rc = -115 [11756.784522] LustreError: 298387:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17929-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11761.938307] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11767.564256] Lustre: server umount fs17929-MDT0001 complete [11793.900740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11793.914904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.926398] Lustre: Skipped 3 previous similar messages [11798.564598] Lustre: server umount lustre-OST0000 complete [11809.186212] Lustre: server umount lustre-MDT0000 complete [11813.323178] LustreError: 290165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151401 with bad export cookie 14548883476555541013 [11813.325646] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.333170] LustreError: 290165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11813.614063] Lustre: server umount lustre-MDT0001 complete [11825.806593] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11828.962066] Key type lgssc unregistered [11829.352650] LNet: 300734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11829.365156] LNetError: 300734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11829.392779] LNet: Removed LNI 192.168.204.114@tcp [11830.416160] Key type .llcrypt unregistered [11830.421921] Key type ._llcrypt unregistered [11851.359234] Key type ._llcrypt registered [11851.362216] Key type .llcrypt registered [11851.494296] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11852.783637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11852.855343] alg: No test for adler32 (adler32-zlib) [11853.952798] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11854.153085] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11855.792316] Key type lgssc registered [11856.820373] Lustre: Echo OBD driver; http://www.lustre.org/ [11866.092660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11866.115982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.706021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.118359] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11877.864179] LustreError: 302025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.894320] LustreError: 302025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11882.155554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.523406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11887.015775] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11895.898804] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11903.071820] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11911.113196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11911.562304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11912.558750] LustreError: 303853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.794129] LustreError: 303855:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11918.080195] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11922.920254] LustreError: 303856:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.206330] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11928.046108] LustreError: 303931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.084740] LustreError: 303931:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11938.284489] LustreError: 303856:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.308420] LustreError: 303856:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11946.314646] loop1: detected capacity change from 0 to 204800000 [11946.504134] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.514548] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.522464] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11952.484671] loop1: detected capacity change from 0 to 204800000 [11954.437813] LustreError: 303853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.451173] LustreError: 303853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [11958.868882] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11959.083382] Lustre: fs26414-OST0000: new disk, initializing [11959.088307] Lustre: srv-fs26414-OST0000: No data found on store. Initialize space. [11959.096574] Lustre: 305760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11959.217399] Lustre: fs26414-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11966.443185] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11976.502671] Lustre: server umount fs26414-OST0000 complete [11989.485327] LustreError: 303855:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11989.526239] LustreError: 303855:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [11990.975635] loop1: detected capacity change from 0 to 204800000 [11994.612494] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11997.904911] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11998.074071] Lustre: fs26414-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12005.105270] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12009.974687] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12015.029148] loop2: detected capacity change from 0 to 204800000 [12015.280447] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12015.293986] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12015.327801] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12021.090097] loop2: detected capacity change from 0 to 204800000 [12027.734220] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12027.875839] LustreError: fs26414-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12027.896678] LustreError: fs26414-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12027.903213] LustreError: 308805:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12027.909386] LustreError: 308805:0:(tgt_mount.c:2129:server_put_super()) no obd fs26414-OST0001 [12027.923341] LustreError: 308805:0:(tgt_mount.c:116:server_deregister_mount()) fs26414-OST0001 not registered [12027.970958] Lustre: server umount fs26414-OST0001 complete [12027.977925] LustreError: 308805:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12039.188536] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12039.362982] Lustre: fs26414-OST0001: new disk, initializing [12039.365991] Lustre: srv-fs26414-OST0001: No data found on store. Initialize space. [12039.372095] Lustre: 309358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12039.451480] Lustre: fs26414-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12047.003631] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12055.666263] Lustre: server umount fs26414-OST0000 complete [12056.038765] LustreError: 303856:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.053852] LustreError: 303856:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 37 previous similar messages [12065.186752] Lustre: server umount fs26414-OST0001 complete [12078.473884] loop1: detected capacity change from 0 to 204800000 [12084.333543] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12084.617042] Lustre: fs26414-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12091.714364] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12099.701477] loop2: detected capacity change from 0 to 204800000 [12106.611952] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12113.976823] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12125.313656] Lustre: server umount fs26414-OST0000 complete [12136.533911] Lustre: server umount fs26414-OST0001 complete [12148.202144] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12153.316612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12153.323580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12153.336246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12153.341164] Lustre: Skipped 1 previous similar message [12156.763058] Lustre: server umount lustre-OST0000 complete [12163.557315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12163.560907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12163.575279] Lustre: Skipped 2 previous similar messages [12163.587747] Lustre: Skipped 1 previous similar message [12168.686801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12168.696515] Lustre: Skipped 1 previous similar message [12173.797185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12173.806790] Lustre: Skipped 1 previous similar message [12174.304454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12174.558580] Lustre: server umount lustre-MDT0000 complete [12179.208550] LustreError: 302004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151767 with bad export cookie 11609241406587158404 [12179.209469] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12179.218536] LustreError: 302004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12179.546688] Lustre: server umount lustre-MDT0001 complete [12191.576350] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12194.390481] Key type lgssc unregistered [12194.693664] LNet: 315285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12194.703757] LNetError: 315285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12194.724754] LNet: Removed LNI 192.168.204.114@tcp [12195.635173] Key type .llcrypt unregistered [12195.639156] Key type ._llcrypt unregistered [12228.795524] Key type ._llcrypt registered [12228.800284] Key type .llcrypt registered [12228.922613] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [12244.732246] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12246.106978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12246.163062] alg: No test for adler32 (adler32-zlib) [12247.195232] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [12247.503740] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12249.240309] Key type lgssc registered [12250.685532] Lustre: Echo OBD driver; http://www.lustre.org/ [12288.446242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12288.466959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.756860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12289.840358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12289.964961] Lustre: lustre-MDT0000: new disk, initializing [12290.112494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12290.144975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12295.317225] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12306.422994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12306.661407] Lustre: 319143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12306.713382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12306.717194] Lustre: Skipped 1 previous similar message [12306.773740] Lustre: lustre-MDT0001: new disk, initializing [12306.856903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12306.906702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12306.925265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12311.555234] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12321.574852] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12327.843419] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12335.650657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.026403] Lustre: lustre-OST0000: new disk, initializing [12336.035187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12336.053893] Lustre: 321116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12336.146834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12343.516986] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12345.465428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12345.485787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12345.682396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12354.386252] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12356.760892] Lustre: Mounted lustre-client - version 2.17.57_137_gfce00d1 [12357.882133] Lustre: Unmounted lustre-client [12371.427252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12371.430713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12371.437818] LustreError: Skipped 1 previous similar message [12371.440447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12371.462709] Lustre: Skipped 1 previous similar message [12371.492718] Lustre: Skipped 1 previous similar message [12372.449895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12372.460064] Lustre: Skipped 1 previous similar message [12374.508765] Lustre: server umount lustre-OST0000 complete [12382.691276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.702425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12382.712468] Lustre: Skipped 1 previous similar message [12384.945168] Lustre: server umount lustre-MDT0000 complete [12389.006575] LustreError: 319135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151976 with bad export cookie 7437821460477296953 [12389.012667] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12389.027652] LustreError: 319135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12389.360346] Lustre: server umount lustre-MDT0001 complete [12398.478265] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12401.095311] Key type lgssc unregistered [12416.738115] LNet: 323277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12416.745412] LNetError: 323277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12486.376911] LNet: Removed LNI 192.168.204.114@tcp [12487.375650] Key type .llcrypt unregistered [12487.379699] Key type ._llcrypt unregistered [12508.497234] Key type ._llcrypt registered [12508.500481] Key type .llcrypt registered [12508.637357] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [12529.858704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12531.855378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12532.148148] alg: No test for adler32 (adler32-zlib) [12533.481918] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [12533.927631] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12535.721220] Key type lgssc registered [12537.616631] Lustre: Echo OBD driver; http://www.lustre.org/ [12572.974809] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:42:39 (1789152159) [12580.547222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12580.565536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.938511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12582.022346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12582.138325] Lustre: lustre-MDT0000: new disk, initializing [12582.324385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12582.375673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12588.227730] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12602.163563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12602.345924] Lustre: 326507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12602.455445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12602.465723] Lustre: Skipped 1 previous similar message [12602.571894] Lustre: lustre-MDT0001: new disk, initializing [12602.734904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12602.780106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12602.799309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12610.044811] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12623.481648] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12630.286917] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12637.441419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12637.896936] Lustre: lustre-OST0000: new disk, initializing [12637.903575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12637.912823] Lustre: 328475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12638.046278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12645.171496] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12645.951888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12645.972098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12646.091288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12656.838573] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12677.092599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12677.092836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.107035] LustreError: Skipped 1 previous similar message [12677.145278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12681.712917] Lustre: server umount lustre-OST0000 complete [12687.341030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12687.353114] Lustre: Skipped 1 previous similar message [12687.361717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12687.367824] Lustre: Skipped 2 previous similar messages [12692.590853] Lustre: server umount lustre-MDT0000 complete [12696.500451] LustreError: 326498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152284 with bad export cookie 16635381673406955427 [12696.509656] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12696.510778] LustreError: 326498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12696.910326] Lustre: server umount lustre-MDT0001 complete [12708.936275] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12711.498994] Key type lgssc unregistered [12711.792610] LNet: 330387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12711.813238] LNetError: 330387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12712.871236] LNet: Removed LNI 192.168.204.114@tcp [12713.793569] Key type .llcrypt unregistered [12713.795958] Key type ._llcrypt unregistered [12735.282528] Key type ._llcrypt registered [12735.287971] Key type .llcrypt registered [12735.419751] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12736.250596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12736.274776] alg: No test for adler32 (adler32-zlib) [12737.370969] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [12737.621038] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12739.328758] Key type lgssc registered [12740.446406] Lustre: Echo OBD driver; http://www.lustre.org/ [12750.569463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12750.637086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.652754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12758.745558] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12762.085921] LustreError: 331663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12762.112400] LustreError: 331663:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12767.212441] LustreError: 331664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12772.333401] LustreError: 331663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12772.980587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.326134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12780.210731] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12790.432457] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12796.972277] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12805.376581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12805.839319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12813.282124] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12821.499170] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12832.316912] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12834.138213] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12839.696517] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [12839.908877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12839.914957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12842.978289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12842.980254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12842.981938] Lustre: Skipped 1 previous similar message [12842.996437] Lustre: Skipped 1 previous similar message [12844.800659] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [12844.812600] Lustre: Skipped 2 previous similar messages [12845.387326] Lustre: server umount lustre-MDT0000 complete [12848.098370] LustreError: 331663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12848.133699] LustreError: 331663:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12850.177969] LustreError: 331645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152438 with bad export cookie 8110764737922331802 [12850.194308] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12850.203803] LustreError: 331645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12853.220124] LustreError: 331658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12853.232797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12853.238277] LustreError: 331658:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12853.268818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12856.924197] Lustre: server umount lustre-MDT0001 complete [12861.565433] Lustre: server umount lustre-OST0000 complete [12871.399991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12876.259947] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12879.512641] Lustre: 335850:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.114@tcp [12882.325308] Lustre: server umount MGS complete [12892.338156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12893.069776] LustreError: 336422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12893.095835] LustreError: 336422:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12893.244340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12899.205100] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12909.189929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12909.475661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12914.318771] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12924.937277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12925.546856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12932.908971] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12941.858398] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12944.028543] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12952.842620] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12955.615745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12961.785385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12961.800743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12961.838534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12961.843903] Lustre: Skipped 2 previous similar messages [12963.578242] Lustre: server umount lustre-OST0000 complete [12969.371861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12969.409206] Lustre: Skipped 1 previous similar message [12974.572071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12974.582624] Lustre: Skipped 3 previous similar messages [12976.032943] Lustre: server umount lustre-MDT0000 complete [12979.688967] LustreError: 336418:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12979.711379] LustreError: 336418:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [12980.519148] Lustre: 330974:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789152552/real 1789152552] req@00000000a6a38fe4 x1876062188407424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789152568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12980.571739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.734752] Lustre: 330975:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789152553/real 1789152553] req@00000000362e6a6d x1876062188407808/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789152569 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12981.748488] LustreError: 336402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152569 with bad export cookie 8110764737922332642 [12981.816930] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12982.457456] Lustre: server umount lustre-MDT0001 complete [12996.334885] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12999.636359] Key type lgssc unregistered [12999.959210] LNet: 339863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12999.975381] LNetError: 339863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13001.003468] LNet: Removed LNI 192.168.204.114@tcp [13002.175956] Key type .llcrypt unregistered [13002.179164] Key type ._llcrypt unregistered [13024.899991] Key type ._llcrypt registered [13024.902238] Key type .llcrypt registered [13025.106557] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [13046.450885] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [13047.698968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13047.763088] alg: No test for adler32 (adler32-zlib) [13048.778688] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [13048.976251] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [13050.632165] Key type lgssc registered [13051.780561] Lustre: Echo OBD driver; http://www.lustre.org/ [13093.123509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13093.208489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13094.477179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13094.518829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13094.635508] Lustre: lustre-MDT0000: new disk, initializing [13094.734185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13094.765810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13099.831133] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13111.055412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13111.255046] Lustre: 343030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13111.325958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13111.330924] Lustre: Skipped 1 previous similar message [13111.490789] Lustre: lustre-MDT0001: new disk, initializing [13111.635449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13111.699884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13111.719748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13116.204583] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13127.434943] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13135.082174] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13142.479281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13142.707848] Lustre: lustre-OST0000: new disk, initializing [13142.711532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13142.717838] Lustre: 344997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13142.857168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13150.334263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13150.347382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13150.494722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13150.703653] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13161.723840] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13167.360086] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13167.778930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13173.604047] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13173.885042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13176.294400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13176.296099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.298205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13176.307474] LustreError: Skipped 1 previous similar message [13176.338990] Lustre: Skipped 1 previous similar message [13181.422578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13181.438576] Lustre: Skipped 2 previous similar messages [13182.325644] Lustre: server umount lustre-OST0000 complete [13186.530313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13186.543986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13186.546327] Lustre: Skipped 1 previous similar message [13186.553217] Lustre: Skipped 1 previous similar message [13191.650684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13191.665333] Lustre: Skipped 1 previous similar message [13192.614533] Lustre: server umount lustre-MDT0000 complete [13196.850319] LustreError: 343022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152784 with bad export cookie 9375436610492672336 [13196.851652] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13196.859053] LustreError: 343022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13197.371624] Lustre: server umount lustre-MDT0001 complete [13207.789635] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:53:14 (1789152794) [13225.200619] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [13236.009627] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [13276.098807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13276.382667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13276.430671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13276.545335] Lustre: lustre-MDT0000: new disk, initializing [13276.635130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13276.664536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13281.946505] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13293.493938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13293.604679] Lustre: 349761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13293.686203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13293.693484] Lustre: Skipped 1 previous similar message [13293.780457] Lustre: lustre-MDT0001: new disk, initializing [13293.890039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13293.938525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13293.955142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13299.601489] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13310.854029] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13318.081741] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13325.704885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13326.018897] Lustre: lustre-OST0000: new disk, initializing [13326.022759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13326.030288] Lustre: 351728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13326.119318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13331.515079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13331.532383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13331.688287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13332.289861] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13342.996125] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13383.144509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13383.148659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13383.162391] LustreError: Skipped 1 previous similar message [13383.165551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13383.210526] Lustre: Skipped 1 previous similar message [13389.421534] Lustre: server umount lustre-OST0000 complete [13396.449398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13396.462460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13396.485313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13396.493443] Lustre: Skipped 3 previous similar messages [13400.611092] Lustre: server umount lustre-MDT0000 complete [13403.620312] LustreError: 352598:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13403.645269] LustreError: 352598:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13405.255650] LustreError: 349753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152993 with bad export cookie 9375436610492673218 [13405.257031] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13405.264979] LustreError: 349753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13405.752650] Lustre: server umount lustre-MDT0001 complete [13420.464037] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13423.852381] Key type lgssc unregistered [13424.270730] LNet: 353908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13424.292477] LNetError: 353908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13424.326979] LNet: Removed LNI 192.168.204.114@tcp [13425.275548] Key type .llcrypt unregistered [13425.277119] Key type ._llcrypt unregistered [13541.746356] Key type ._llcrypt registered [13541.752561] Key type .llcrypt registered [13541.907295] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13542.350310] Key type .llcrypt unregistered [13542.354843] Key type ._llcrypt unregistered [13633.443329] Key type ._llcrypt registered [13633.444970] Key type .llcrypt registered [13633.577663] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13633.937362] Key type .llcrypt unregistered [13633.939184] Key type ._llcrypt unregistered [13667.436201] Key type ._llcrypt registered [13667.441255] Key type .llcrypt registered [13667.568346] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13667.924277] Key type .llcrypt unregistered [13667.927985] Key type ._llcrypt unregistered [13701.476377] Key type ._llcrypt registered [13701.482807] Key type .llcrypt registered [13701.672711] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13702.172325] Key type .llcrypt unregistered [13702.178150] Key type ._llcrypt unregistered [13784.238734] Key type ._llcrypt registered [13784.240257] Key type .llcrypt registered [13784.392445] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13784.790886] Key type .llcrypt unregistered [13784.805754] Key type ._llcrypt unregistered [13820.085960] Key type ._llcrypt registered [13820.092401] Key type .llcrypt registered [13820.246146] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13820.673330] Key type .llcrypt unregistered [13820.681692] Key type ._llcrypt unregistered [13859.540091] Key type ._llcrypt registered [13859.549056] Key type .llcrypt registered [13859.759183] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13860.368047] Key type .llcrypt unregistered [13860.369837] Key type ._llcrypt unregistered [13896.433664] Key type ._llcrypt registered [13896.439118] Key type .llcrypt registered [13896.596279] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13896.905420] Key type .llcrypt unregistered [13896.909984] Key type ._llcrypt unregistered [13928.659434] Key type ._llcrypt registered [13928.662188] Key type .llcrypt registered [13928.767747] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13929.152053] Key type .llcrypt unregistered [13929.155636] Key type ._llcrypt unregistered [13967.546257] Key type ._llcrypt registered [13967.549249] Key type .llcrypt registered [13967.648517] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13967.952509] Key type .llcrypt unregistered [13967.957560] Key type ._llcrypt unregistered [14045.129494] Key type ._llcrypt registered [14045.133776] Key type .llcrypt registered [14045.351671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14045.766295] Key type .llcrypt unregistered [14045.770714] Key type ._llcrypt unregistered [14083.442824] Key type ._llcrypt registered [14083.446260] Key type .llcrypt registered [14083.578268] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14084.092647] Key type .llcrypt unregistered [14084.095185] Key type ._llcrypt unregistered [14117.732603] Key type ._llcrypt registered [14117.738338] Key type .llcrypt registered [14117.887859] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14118.294633] Key type .llcrypt unregistered [14118.296426] Key type ._llcrypt unregistered [14186.217542] Key type ._llcrypt registered [14186.220152] Key type .llcrypt registered [14186.339634] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [14204.063686] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14205.255535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14205.304345] alg: No test for adler32 (adler32-zlib) [14206.474835] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [14206.731236] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [14208.416596] Key type lgssc registered [14209.667711] Lustre: Echo OBD driver; http://www.lustre.org/ [14251.915873] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14251.958948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14253.319484] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14253.360441] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14253.572122] Lustre: tst250fs-MDT0000: new disk, initializing [14253.682667] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14253.721857] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14259.322317] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14272.263524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14272.462484] Lustre: 374337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14272.604044] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14272.613446] Lustre: Skipped 1 previous similar message [14272.747992] Lustre: tst250fs-MDT0001: new disk, initializing [14272.918305] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14273.010183] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14273.023700] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14279.126488] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14284.694952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14293.503843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14294.020911] Lustre: tst250fs-OST0000: new disk, initializing [14294.027539] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14294.037630] Lustre: 375974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14294.188827] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14296.535835] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14296.548749] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14296.705951] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14301.706227] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14314.640056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14314.882603] Lustre: tst250fs-OST0001: new disk, initializing [14314.888627] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14314.898253] Lustre: 376844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14315.023048] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14322.440174] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14323.370205] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14323.388161] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14323.490336] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14328.985422] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14331.662129] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14331.676118] Lustre: Skipped 2 previous similar messages [14354.918571] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14354.937329] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14354.955986] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14359.522045] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14359.538401] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14359.556277] Lustre: Skipped 2 previous similar messages [14359.578651] Lustre: Skipped 3 previous similar messages [14361.201564] Lustre: server umount tst250fs-MDT0000 complete [14365.912362] LustreError: 375134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789153953 with bad export cookie 12197928986928366398 [14365.917070] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14365.925415] LustreError: 375134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14366.405802] Lustre: server umount tst250fs-MDT0001 complete [14380.485884] Lustre: server umount tst250fs-OST0000 complete [14395.149512] Lustre: server umount tst250fs-OST0001 complete [14416.784691] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [14426.064546] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14462.971459] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:14:09 (1789154049) [14465.341932] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14487.124733] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [14501.850031] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14539.775295] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14268 sec ========= 15:15:26 (1789154126) [14542.041677] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:15:28 (1789154128) === [14547.084355] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:15:33 (1789154133) === [14597.702991] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14602.745782] Key type lgssc unregistered [14603.281311] LNet: 386011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14603.305531] LNetError: 386011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14603.367900] LNet: Removed LNI 192.168.204.114@tcp [14604.797233] Key type .llcrypt unregistered [14604.799025] Key type ._llcrypt unregistered