[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527987083 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003200] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008349] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012047] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.014058] *** VALIDATE selinux *** [ 0.022094] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026423] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] *** VALIDATE tmpfs *** [ 0.031337] *** VALIDATE proc *** [ 0.032000] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.034248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.041085] debug: unmapping init [mem 0xffffffff94a59000-0xffffffff94a60fff] [ 0.043171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.052293] rcu: Hierarchical SRCU implementation. [ 0.054467] smp: Bringing up secondary CPUs ... [ 0.055559] x86: Booting SMP configuration: [ 0.056015] .... node #0, CPUs: #1 #2 #3 [ 0.063061] smp: Brought up 1 node, 4 CPUs [ 0.065010] smpboot: Max logical packages: 1 [ 0.066011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101075] node 0 deferred pages initialised in 33ms [ 0.105034] devtmpfs: initialized [ 0.106341] x86/mm: Memory block size: 128MB [ 0.109473] gcov: version magic: 0x41383552 [ 0.111367] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114025] pinctrl core: initialized pinctrl subsystem [ 0.115341] [ 0.115945] ************************************************************* [ 0.116000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120013] ** ** [ 0.122009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125009] ** ** [ 0.127009] ** This means that this kernel is built to expose internal ** [ 0.129009] ** IOMMU data structures, which may compromise security on ** [ 0.130005] ** your system. ** [ 0.132008] ** ** [ 0.135010] ** If you see this message and you are not debugging the ** [ 0.137008] ** kernel, report this immediately to your vendor! ** [ 0.139010] ** ** [ 0.142010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152033] ************************************************************* [ 0.162020] NET: Registered protocol family 16 [ 0.163529] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181037] cpuidle: using governor menu [ 0.184000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192044] PCI: Using configuration type 1 for base access [ 0.199169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218024] cryptd: max_cpu_qlen set to 1000 [ 0.219412] ACPI: Added _OSI(Module Device) [ 0.220000] ACPI: Added _OSI(Processor Device) [ 0.220000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228104] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232553] ACPI: Interpreter enabled [ 0.233053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234021] ACPI: Using IOAPIC for interrupt routing [ 0.235074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238248] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241520] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254000] acpiphp: Slot [2] registered [ 0.254000] acpiphp: Slot [5] registered [ 0.256077] acpiphp: Slot [6] registered [ 0.257074] acpiphp: Slot [7] registered [ 0.258071] acpiphp: Slot [8] registered [ 0.260074] acpiphp: Slot [9] registered [ 0.261096] acpiphp: Slot [10] registered [ 0.263067] acpiphp: Slot [3] registered [ 0.264000] acpiphp: Slot [4] registered [ 0.264000] acpiphp: Slot [11] registered [ 0.264064] acpiphp: Slot [12] registered [ 0.265000] acpiphp: Slot [13] registered [ 0.266100] acpiphp: Slot [14] registered [ 0.267051] acpiphp: Slot [15] registered [ 0.268005] acpiphp: Slot [16] registered [ 0.270095] acpiphp: Slot [17] registered [ 0.271147] acpiphp: Slot [18] registered [ 0.273064] acpiphp: Slot [19] registered [ 0.274286] acpiphp: Slot [20] registered [ 0.276089] acpiphp: Slot [21] registered [ 0.278131] acpiphp: Slot [22] registered [ 0.280164] acpiphp: Slot [23] registered [ 0.282126] acpiphp: Slot [24] registered [ 0.284108] acpiphp: Slot [25] registered [ 0.286108] acpiphp: Slot [26] registered [ 0.288071] acpiphp: Slot [27] registered [ 0.289085] acpiphp: Slot [28] registered [ 0.291066] acpiphp: Slot [29] registered [ 0.292059] acpiphp: Slot [30] registered [ 0.293044] acpiphp: Slot [31] registered [ 0.293964] PCI host bridge to bus 0000:00 [ 0.295013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307020] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.308676] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311190] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320758] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334544] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.342627] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.368012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446682] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494939] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.502016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.530016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.541899] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.549012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.555013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.601145] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.629013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640440] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.658017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.675015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.686000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.689358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690257] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.692515] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.695302] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.701145] iommu: Default domain type: Passthrough [ 0.703397] SCSI subsystem initialized [ 0.704090] ACPI: bus type USB registered [ 0.705092] usbcore: registered new interface driver usbfs [ 0.708052] usbcore: registered new interface driver hub [ 0.710065] usbcore: registered new device driver usb [ 0.711197] pps_core: LinuxPPS API ver. 1 registered [ 0.713008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718065] PTP clock support registered [ 0.720126] EDAC MC: Ver: 3.0.0 [ 0.722185] PCI: Using ACPI for IRQ routing [ 0.724197] NetLabel: Initializing [ 0.725009] NetLabel: domain hash size = 128 [ 0.728011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.731082] NetLabel: unlabeled traffic allowed by default [ 0.734120] vgaarb: loaded [ 0.736391] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.738012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.747382] clocksource: Switched to clocksource kvm-clock [ 0.855784] VFS: Disk quotas dquot_6.6.0 [ 0.857080] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.859419] *** VALIDATE ramfs *** [ 0.860688] *** VALIDATE hugetlbfs *** [ 0.862245] pnp: PnP ACPI init [ 0.864926] pnp: PnP ACPI: found 6 devices [ 0.884110] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.887378] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888922] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.891750] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893793] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.896191] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.898614] NET: Registered protocol family 2 [ 0.900218] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.903334] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.905329] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.908706] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910536] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912086] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.913982] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.915621] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917558] NET: Registered protocol family 1 [ 0.919100] RPC: Registered named UNIX socket transport module. [ 0.920180] RPC: Registered udp transport module. [ 0.921051] RPC: Registered tcp transport module. [ 0.921920] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.923028] NET: Registered protocol family 44 [ 0.923850] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.925099] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933992] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.937210] PCI: CLS 0 bytes, default 64 [ 0.940881] Unpacking initramfs... [ 2.981823] debug: unmapping init [mem 0xffff98113cc54000-0xffff98113ffbffff] [ 2.988556] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.990524] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.994389] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.454905] hrtimer: interrupt took 19238998 ns [ 3.866613] Initialise system trusted keyrings [ 3.868142] Key type blacklist registered [ 3.869774] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.878465] zbud: loaded [ 3.881498] *** VALIDATE nfs *** [ 3.882537] *** VALIDATE nfs4 *** [ 3.884087] pstore: using deflate compression [ 3.887843] Platform Keyring initialized [ 4.133733] NET: Registered protocol family 38 [ 4.136273] Key type asymmetric registered [ 4.138328] Asymmetric key parser 'x509' registered [ 4.140746] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.291513] io scheduler mq-deadline registered [ 4.697375] io scheduler kyber registered [ 4.699367] io scheduler bfq registered [ 5.413627] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.416123] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.419227] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.446145] ACPI: Power Button [PWRF] [ 5.452802] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.459450] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.477814] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.489519] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.531990] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.560428] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.588289] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.592164] Non-volatile memory driver v1.3 [ 5.593264] Linux agpgart interface v0.103 [ 5.624126] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 5.626649] vda: detected capacity change from 0 to 74645504 [ 5.643748] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.646016] vdb: detected capacity change from 0 to 1073741824 [ 5.663744] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.666574] vdc: detected capacity change from 0 to 2621440000 [ 5.683871] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.686568] vdd: detected capacity change from 0 to 2621440000 [ 5.704578] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.707229] vde: detected capacity change from 0 to 4294967296 [ 5.727684] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.730579] vdf: detected capacity change from 0 to 4294967296 [ 5.737076] libphy: Fixed MDIO Bus: probed [ 5.742990] usbcore: registered new interface driver usbserial_generic [ 5.745711] usbserial: USB Serial support registered for generic [ 5.747670] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.752411] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.753943] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.756235] mousedev: PS/2 mouse device common for all mice [ 5.758909] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.760307] rtc_cmos 00:05: RTC can wake from S4 [ 5.767979] rtc_cmos 00:05: registered as rtc0 [ 5.769512] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.773056] intel_pstate: CPU model not supported [ 5.776545] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.778565] hid: raw HID events driver (C) Jiri Kosina [ 5.785607] usbcore: registered new interface driver usbhid [ 5.787565] usbhid: USB HID core driver [ 5.789356] drop_monitor: Initializing network drop monitor service [ 5.792017] Initializing XFRM netlink socket [ 5.794296] NET: Registered protocol family 10 [ 5.796916] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.797932] Segment Routing with IPv6 [ 5.802731] NET: Registered protocol family 17 [ 5.804647] mpls_gso: MPLS GSO support [ 5.809854] RAS: Correctable Errors collector initialized. [ 5.811778] AVX version of gcm_enc/dec engaged. [ 5.813325] AES CTR mode by8 optimization enabled [ 5.895462] sched_clock: Marking stable (5895298454, 0)->(6931661915, -1036363461) [ 5.898815] registered taskstats version 1 [ 5.900613] Loading compiled-in X.509 certificates [ 5.902590] zswap: loaded using pool lzo/zbud [ 5.927116] Key type big_key registered [ 5.940051] Key type encrypted registered [ 5.941153] ima: No TPM chip found, activating TPM-bypass! [ 5.942555] ima: Allocated hash algorithm: sha1 [ 5.944110] ima: No architecture policies found [ 5.945932] evm: Initialising EVM extended attributes: [ 5.947844] evm: security.selinux [ 5.948964] evm: security.ima [ 5.949858] evm: security.capability [ 5.951202] evm: HMAC attrs: 0x1 [ 5.953604] rtc_cmos 00:05: setting system clock to 2026-07-18 22:14:47 UTC (1784412887) [ 5.960777] debug: unmapping init [mem 0xffffffff95a03000-0xffffffff95bfffff] [ 5.963848] debug: unmapping init [mem 0xffffffff94782000-0xffffffff94a58fff] [ 5.972125] Write protecting the kernel read-only data: 28672k [ 5.976198] debug: unmapping init [mem 0xffffffff92e03000-0xffffffff92ffffff] [ 5.978917] debug: unmapping init [mem 0xffffffff93714000-0xffffffff937fffff] [ 6.025288] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.035668] systemd[1]: Detected virtualization kvm. [ 6.037684] systemd[1]: Detected architecture x86-64. [ 6.039697] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.065692] systemd[1]: No hostname configured. [ 6.067128] systemd[1]: Set hostname to . [ 6.069338] random: systemd: uninitialized urandom read (16 bytes read) [ 6.071669] systemd[1]: Initializing machine ID from random generator. [ 6.283626] random: ln: uninitialized urandom read (6 bytes read) [ 6.429631] random: systemd: uninitialized urandom read (16 bytes read) [ 6.435643] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.439049] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.444165] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.754717] device-mapper: uevent: version 1.0.3 [ 7.756603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.204858] virtio_net virtio0 ens2: renamed from eth0 [ 9.254408] random: fast init done [ 9.695840] scsi host0: ata_piix [ 9.779911] scsi host1: ata_piix [ 9.782223] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.790477] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.482459] random: crng init done [ 14.487987] random: 7 urandom warning(s) missed due to ratelimiting [ 16.860959] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.534297] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.750256] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.160562] SELinux: Disabled at runtime. [ 21.271525] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.276686] systemd[1]: Detected virtualization kvm. [ 21.277863] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.925637] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.933573] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.949232] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.954877] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.965331] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.989465] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.996297] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 23.683568] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 24.695820] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.354075] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.739621] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.683832] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.737555] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.549494] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.390357] NFS: Registering the id_resolver key type [ 31.392618] Key type id_resolver registered [ 31.394588] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg117-server login: [ 60.660230] libcfs: loading out-of-tree module taints kernel. [ 60.679405] Key type ._llcrypt registered [ 60.680715] Key type .llcrypt registered [ 60.720224] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 68.006213] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 68.585092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.592028] alg: No test for adler32 (adler32-zlib) [ 69.559398] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 69.863493] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 71.480179] Key type lgssc registered [ 72.075805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.182173] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.767541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 99.711618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 99.724539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.818932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 100.830476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 100.865771] Lustre: lustre-MDT0000: new disk, initializing [ 100.888148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.894120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 102.269522] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.615152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 107.647817] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 107.658128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 107.660407] Lustre: Skipped 1 previous similar message [ 107.694161] Lustre: lustre-MDT0001: new disk, initializing [ 107.714311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.722759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 107.727106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 109.144956] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.502215] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 114.960666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 115.048345] Lustre: lustre-OST0000: new disk, initializing [ 115.050668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 115.053408] Lustre: 8419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 115.071539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 116.590872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 116.593950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 116.624299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 117.066620] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.889514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 122.947918] Lustre: lustre-OST0001: new disk, initializing [ 122.950200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 122.953309] Lustre: 9494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 122.978095] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 124.974122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 124.978248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 124.996490] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 125.354288] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.294443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 134.057341] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 136.308116] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing check_logdir /tmp/testlogs/ [ 137.935428] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing yml_node [ 139.811697] Lustre: DEBUG MARKER: Client: 2.17.54.173 [ 140.773634] Lustre: DEBUG MARKER: MDS: 2.17.54.173 [ 141.653708] Lustre: DEBUG MARKER: OSS: 2.17.54.173 [ 142.205609] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 18 18:17:03 EDT 2026 [ 147.986173] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 148.549308] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 160.736496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 160.737275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 160.740578] Lustre: Skipped 3 previous similar messages [ 160.743416] Lustre: Skipped 2 previous similar messages [ 165.856565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.860688] Lustre: Skipped 3 previous similar messages [ 166.560312] Lustre: server umount lustre-MDT0000 complete [ 168.027563] LustreError: 7435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413049 with bad export cookie 16631395331775693053 [ 168.028296] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 168.032194] LustreError: 7435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 170.976680] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 170.976886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 170.981661] Lustre: Skipped 1 previous similar message [ 173.984507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.986157] Lustre: Skipped 2 previous similar messages [ 174.183767] Lustre: server umount lustre-MDT0001 complete [ 181.636052] Lustre: server umount lustre-OST0000 complete [ 189.146097] Lustre: server umount lustre-OST0001 complete [ 191.338457] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 194.614649] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 208.228145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.315764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.326579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.365213] Lustre: lustre-MDT0000: new disk, initializing [ 208.400727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.414595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.940760] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 214.497913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.546511] Lustre: 15034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.660786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.159270] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 220.518645] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 222.616076] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 225.255120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.364341] Lustre: lustre-OST0000: new disk, initializing [ 225.365518] Lustre: Skipped 1 previous similar message [ 225.366968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.369235] Lustre: Skipped 2 previous similar messages [ 225.371271] Lustre: 17001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.400401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.403275] Lustre: Skipped 1 previous similar message [ 226.681858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.685321] Lustre: Skipped 1 previous similar message [ 226.687977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.725373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.448120] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 231.219704] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 232.681522] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 232.758117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 234.209332] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 234.301144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 237.025962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 237.027066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.028994] LustreError: Skipped 1 previous similar message [ 237.033400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 237.037334] Lustre: Skipped 1 previous similar message [ 247.267559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 247.269749] Lustre: Skipped 3 previous similar messages [ 249.312136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 249.402330] Lustre: server umount lustre-OST0000 complete [ 250.848986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 250.852229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.856178] Lustre: Skipped 1 previous similar message [ 265.184136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 265.260682] Lustre: server umount lustre-MDT0000 complete [ 266.761309] LustreError: 15028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413148 with bad export cookie 16631395331775694509 [ 266.764031] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.764610] LustreError: 15028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 266.894130] Lustre: server umount lustre-MDT0001 complete [ 268.999869] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:19:10 (1784413150) [ 276.084035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.242613] LustreError: 19875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.253294] LustreError: 19875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 276.271616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 277.831453] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 281.345661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.133164] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 286.398728] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 288.818503] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 289.762585] Lustre: Failing over lustre-MDT0000 [ 289.877921] Lustre: server umount lustre-MDT0000 complete [ 290.786211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.787910] LustreError: 19871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.791780] Lustre: Skipped 2 previous similar messages [ 291.809336] LustreError: 19894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.814764] LustreError: 19894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 296.931861] LustreError: 19875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.938777] LustreError: 19875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 302.049151] LustreError: 19876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.055240] LustreError: 19876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 303.774399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.851735] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.996434] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 307.044877] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 307.048227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 307.067603] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 310.609353] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 317.410604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 317.414377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.423489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.426954] Lustre: Skipped 7 previous similar messages [ 327.136297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 327.210907] Lustre: server umount lustre-MDT0000 complete [ 328.561139] LustreError: 20610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413210 with bad export cookie 16631395331775695930 [ 328.564194] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.565307] LustreError: 20610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 328.699859] Lustre: server umount lustre-MDT0001 complete [ 331.827698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 331.980348] LustreError: 23207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.010255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.013444] Lustre: Skipped 2 previous similar messages [ 333.421647] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 336.525563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.127721] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 340.948260] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 343.083074] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 343.852932] Lustre: Failing over lustre-MDT0000 [ 343.950515] Lustre: server umount lustre-MDT0000 complete [ 346.913356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.917644] Lustre: Skipped 2 previous similar messages [ 352.224961] LustreError: 24566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.229241] LustreError: 24566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 357.096590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.137257] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.668730] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 362.337929] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 362.340888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 362.343488] Lustre: Skipped 1 previous similar message [ 362.349161] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 362.595984] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 364.523968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.525560] Lustre: Skipped 6 previous similar messages [ 370.656636] Lustre: server umount lustre-MDT0000 complete [ 371.933412] LustreError: 23189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413253 with bad export cookie 16631395331775696868 [ 371.935945] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.938052] LustreError: 23189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 372.051921] Lustre: server umount lustre-MDT0001 complete [ 375.087935] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:20:56 (1784413256) [ 380.552294] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 384.039515] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 396.587156] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 399.961318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.035948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.044498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 400.075726] Lustre: lustre-MDT0000: new disk, initializing [ 400.094609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.097904] Lustre: Skipped 2 previous similar messages [ 400.103606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.485222] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 405.635415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.688321] Lustre: 29568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 405.786651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 407.185451] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 410.817469] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.585140] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 414.774898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.868981] Lustre: 31538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 416.193602] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 416.214374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 416.220377] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 416.716452] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 420.233407] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 422.066703] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 426.469604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 426.470451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.471698] LustreError: Skipped 1 previous similar message [ 426.476131] Lustre: Skipped 3 previous similar messages [ 431.587877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 431.590239] Lustre: Skipped 5 previous similar messages [ 431.793693] Lustre: server umount lustre-OST0000 complete [ 436.705745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 439.191754] Lustre: server umount lustre-MDT0000 complete [ 440.470213] LustreError: 29561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413322 with bad export cookie 16631395331775697274 [ 440.473731] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 440.474092] LustreError: 29561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 440.650645] Lustre: server umount lustre-MDT0001 complete [ 444.246399] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 445.245750] Key type lgssc unregistered [ 445.372321] LNet: 33490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.374328] LNetError: 33490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 445.381268] LNet: Removed LNI 192.168.201.117@tcp [ 445.702103] Key type .llcrypt unregistered [ 445.703166] Key type ._llcrypt unregistered [ 454.607547] Key type ._llcrypt registered [ 454.608561] Key type .llcrypt registered [ 454.648797] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 454.982302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 454.996830] alg: No test for adler32 (adler32-zlib) [ 455.862074] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 455.951583] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 457.536157] Key type lgssc registered [ 457.916857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 460.581397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 460.589607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.722047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.025557] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 465.751678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.886771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 467.195458] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 469.600267] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.417365] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 473.796048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.947607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.009213] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 476.039738] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 478.584745] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 493.904378] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:22:55 (1784413375) [ 494.579941] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 495.235851] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 497.531877] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:22:58 (1784413378) [ 500.445855] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:23:01 (1784413381) [ 500.811837] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 503.761141] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 503.764794] Lustre: Skipped 2 previous similar messages [ 504.354360] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:23:05 (1784413385) [ 507.877649] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 507.879819] Lustre: Skipped 7 previous similar messages [ 512.337136] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:23:13 (1784413393) [ 513.291974] Lustre: *** cfs_fail_loc=131b, val=2*** [ 514.485279] Lustre: *** cfs_fail_loc=131b, val=2*** [ 517.223081] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:23:18 (1784413398) [ 517.922690] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 517.925174] Lustre: Skipped 6 previous similar messages [ 522.696865] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:23:23 (1784413403) [ 527.329498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.333952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.340401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.354526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.354867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.360223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.686836] Lustre: server umount lustre-OST0000 complete [ 537.569025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 537.572209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.576822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.578772] Lustre: Skipped 1 previous similar message [ 543.713247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 543.716076] Lustre: Skipped 3 previous similar messages [ 548.320146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.384383] Lustre: server umount lustre-MDT0000 complete [ 548.833102] LustreError: 34881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.839475] LustreError: 34881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 549.688058] LustreError: 36720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413431 with bad export cookie 3870039721580086763 [ 549.689744] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.691447] LustreError: 36720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 549.827706] Lustre: server umount lustre-MDT0001 complete [ 553.307029] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 554.267721] Key type lgssc unregistered [ 554.387305] LNet: 42720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.389961] LNetError: 42720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.400336] LNet: Removed LNI 192.168.201.117@tcp [ 554.692511] Key type .llcrypt unregistered [ 554.693524] Key type ._llcrypt unregistered [ 560.546354] Key type ._llcrypt registered [ 560.547961] Key type .llcrypt registered [ 560.584378] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 566.782934] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 567.103029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 567.154081] alg: No test for adler32 (adler32-zlib) [ 567.997689] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 568.084517] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 569.656119] Key type lgssc registered [ 570.049609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 581.851607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 581.858526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.945735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 582.956431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 582.991368] Lustre: lustre-MDT0000: new disk, initializing [ 583.012140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.018623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 584.395593] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 588.611348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.654593] Lustre: 45845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 588.682758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 588.684986] Lustre: Skipped 1 previous similar message [ 588.725415] Lustre: lustre-MDT0001: new disk, initializing [ 588.760871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 588.773573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 588.776893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 590.149786] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 593.792323] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 595.585686] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.801094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.906555] Lustre: lustre-OST0000: new disk, initializing [ 597.908195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 597.911904] Lustre: 47813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 597.938769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.383159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 598.386426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 598.415791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 599.752208] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 603.380872] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 604.796061] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 604.868012] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 606.154971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 606.221292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 608.738019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 608.739890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.742243] LustreError: Skipped 1 previous similar message [ 608.747386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 608.751325] Lustre: Skipped 1 previous similar message [ 612.942855] Lustre: server umount lustre-OST0000 complete [ 614.368940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.371075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.374438] Lustre: Skipped 1 previous similar message [ 614.376238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.978052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.978080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.318099] Lustre: server umount lustre-MDT0000 complete [ 621.495226] LustreError: 45837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413503 with bad export cookie 11714924794128057382 [ 621.498512] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.498811] LustreError: 45837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 621.600408] Lustre: server umount lustre-MDT0001 complete [ 624.632581] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:25:05 (1784413505) [ 626.735798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.892113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.225564] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 631.140496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.263905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.621347] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 635.040285] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 636.941832] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 639.373722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.487314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.560194] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 644.201843] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 656.406914] LustreError: 52815:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 656.710540] Lustre: Modifying parameter general.timeout=1 in log params [ 657.949766] Lustre: Modifying parameter general.timeout=5 in log params [ 657.951892] Lustre: Skipped 3 previous similar messages [ 660.018692] Lustre: Modifying parameter general.timeout=12 in log params [ 660.020216] Lustre: Skipped 6 previous similar messages [ 664.207908] Lustre: Modifying parameter general.timeout=26 in log params [ 664.209247] Lustre: Skipped 13 previous similar messages [ 672.445605] Lustre: Modifying parameter general.timeout=54 in log params [ 672.446916] Lustre: Skipped 27 previous similar messages [ 688.662260] Lustre: Modifying parameter general.timeout=109 in log params [ 688.663782] Lustre: Skipped 54 previous similar messages [ 697.889939] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:26:19 (1784413579) [ 698.825100] Lustre: 59708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 700.097473] Lustre: 59921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 700.102621] Lustre: 59921:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 710.616732] Lustre: 61547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 710.621292] Lustre: 61547:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 3 previous similar messages [ 710.628753] LustreError: 61547:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 711.922402] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:26:33 (1784413593) [ 718.304771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.306862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.310206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 718.311689] Lustre: Skipped 1 previous similar message [ 721.621909] Lustre: server umount lustre-MDT0000 complete [ 722.816038] LustreError: 52865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413604 with bad export cookie 11714924794128058250 [ 722.818419] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.820046] LustreError: 52865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 722.923481] Lustre: server umount lustre-MDT0001 complete [ 734.420841] Lustre: server umount lustre-OST0000 complete [ 743.172070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.260903] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 743.271806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 743.275924] Lustre: Skipped 43 previous similar messages [ 743.373169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.783624] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 747.642664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.673648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 747.681719] Lustre: 64069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 749.132336] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 751.596186] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 753.393322] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 755.627931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 755.690960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 757.736686] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 760.099672] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 760.657505] Lustre: 66630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 762.848771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.851035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.853888] Lustre: Skipped 2 previous similar messages [ 762.855465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 762.856884] Lustre: Skipped 3 previous similar messages [ 767.458347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 768.653872] Lustre: server umount lustre-OST0000 complete [ 772.577421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.578125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 772.581721] Lustre: Skipped 2 previous similar messages [ 772.584526] Lustre: Skipped 2 previous similar messages [ 784.352141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.411978] Lustre: server umount lustre-MDT0000 complete [ 785.681405] LustreError: 64809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413667 with bad export cookie 11714924794128060497 [ 785.683783] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.684655] LustreError: 64809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 785.796635] Lustre: server umount lustre-MDT0001 complete [ 789.388791] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 790.347881] Key type lgssc unregistered [ 790.480375] LNet: 67838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.483588] LNetError: 67838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.494341] LNet: Removed LNI 192.168.201.117@tcp [ 790.794692] Key type .llcrypt unregistered [ 790.796517] Key type ._llcrypt unregistered [ 799.590601] Key type ._llcrypt registered [ 799.592292] Key type .llcrypt registered [ 799.637438] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 799.952200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.980473] alg: No test for adler32 (adler32-zlib) [ 800.827307] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 800.906863] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 802.480073] Key type lgssc registered [ 802.804788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.334963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 805.344104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.504367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.890266] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 810.830624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.048915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 812.579877] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 815.038314] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 816.850535] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 819.132222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.319754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.281202] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 823.692823] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 836.876468] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 841.696735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 841.699031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.702547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 842.209897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 842.210013] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.212271] Lustre: Skipped 1 previous similar message [ 842.216684] Lustre: Skipped 1 previous similar message [ 845.267530] Lustre: server umount lustre-MDT0000 complete [ 846.546155] LustreError: 69207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413728 with bad export cookie 3212985752457419263 [ 846.548577] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.550539] LustreError: 69207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 846.665904] Lustre: server umount lustre-MDT0001 complete [ 858.262648] Lustre: server umount lustre-OST0000 complete [ 867.206014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.302898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 867.312791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 867.394055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.753531] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 871.609766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.647060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 871.656748] Lustre: 74109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 871.825102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.201955] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 875.628087] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 877.580315] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 879.908149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 879.972855] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 880.099615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.982491] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 884.305900] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.816123] Lustre: 76682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 885.408756] Lustre: Modifying parameter general.jobid_var=disable in log params [ 888.290069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.291778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.294164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.297852] Lustre: Skipped 1 previous similar message [ 888.299760] Lustre: Skipped 1 previous similar message [ 892.688136] Lustre: server umount lustre-OST0000 complete [ 897.504767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 897.506997] LustreError: Skipped 1 previous similar message [ 897.508258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.512101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.649325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.651500] Lustre: Skipped 3 previous similar messages [ 908.256130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 908.317681] Lustre: server umount lustre-MDT0000 complete [ 909.629950] LustreError: 74855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413791 with bad export cookie 3212985752457420075 [ 909.633324] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.634947] LustreError: 74855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 909.758685] Lustre: server umount lustre-MDT0001 complete [ 913.499452] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 914.541995] Key type lgssc unregistered [ 914.669455] LNet: 77892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.672844] LNetError: 77892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.682313] LNet: Removed LNI 192.168.201.117@tcp [ 915.012108] Key type .llcrypt unregistered [ 915.014032] Key type ._llcrypt unregistered [ 924.905614] Key type ._llcrypt registered [ 924.907149] Key type .llcrypt registered [ 924.945121] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 925.379560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.387489] alg: No test for adler32 (adler32-zlib) [ 926.241789] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 926.335230] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 927.912145] Key type lgssc registered [ 928.286749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 930.873328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 930.879832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.017415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.391302] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 936.236935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.443841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.733983] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 940.105933] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 941.889715] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 944.065926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 944.244844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.172807] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 948.609142] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 961.339375] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 961.972517] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 961.974562] Lustre: Skipped 1 previous similar message [ 963.238136] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 963.239993] Lustre: Skipped 3 previous similar messages [ 965.316912] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 965.318895] Lustre: Skipped 6 previous similar messages [ 969.452906] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 969.455736] Lustre: Skipped 13 previous similar messages [ 977.674791] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 977.676887] Lustre: Skipped 27 previous similar messages [ 994.024111] Lustre: 87365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 994.337819] Lustre: Disabling parameter general.jobid_var= in log params [ 994.339278] Lustre: Skipped 44 previous similar messages [ 994.922605] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:31:16 (1784413876) [ 995.399203] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 995.959457] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:31:17 (1784413877) [ 997.858520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.860919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.864757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.886086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.886599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.890363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.002398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.004734] Lustre: Skipped 2 previous similar messages [ 1007.013051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.014910] Lustre: Skipped 1 previous similar message [ 1011.168103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1011.219785] Lustre: server umount lustre-OST0000 complete [ 1013.216882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1013.220474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.224887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.313821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.579817] Lustre: server umount lustre-MDT0000 complete [ 1019.828051] LustreError: 79264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413901 with bad export cookie 17921547361137650166 [ 1019.831244] LustreError: 79264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1019.831603] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.941336] Lustre: server umount lustre-MDT0001 complete [ 1023.740223] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1024.769637] Key type lgssc unregistered [ 1024.894381] LNet: 88615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.896832] LNetError: 88615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.903304] LNet: Removed LNI 192.168.201.117@tcp [ 1025.225388] Key type .llcrypt unregistered [ 1025.227141] Key type ._llcrypt unregistered [ 1026.809292] Key type ._llcrypt registered [ 1026.810342] Key type .llcrypt registered [ 1026.858104] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1028.397786] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules [ 1028.744646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.796619] alg: No test for adler32 (adler32-zlib) [ 1029.663265] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1029.754368] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1031.336169] Key type lgssc registered [ 1031.677038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1032.052204] LustreError: 89768:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1039.847836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1042.926686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1053.313367] LustreError: 89768:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1053.320529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.477294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.775717] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1058.503051] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:32:19 (1784413939) [ 1060.012548] LustreError: 90823:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1060.015750] LustreError: 90823:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1060.018935] LustreError: 90823:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1060.123939] Lustre: server umount lustre-MDT0000 complete [ 1064.915964] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1065.870477] Key type lgssc unregistered [ 1065.994373] LNet: 92316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.997412] LNetError: 92316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.005279] LNet: Removed LNI 192.168.201.117@tcp [ 1066.283738] Key type .llcrypt unregistered [ 1066.285411] Key type ._llcrypt unregistered [ 1073.379103] Key type ._llcrypt registered [ 1073.380654] Key type .llcrypt registered [ 1073.414369] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1073.819990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1073.828825] alg: No test for adler32 (adler32-zlib) [ 1074.692633] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1074.784832] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1076.376148] Key type lgssc registered [ 1076.751156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1079.295899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1079.301684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.442039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.712581] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1084.630370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.848721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.298933] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1088.811911] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1090.727148] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1092.987651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1093.209836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.050833] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1097.354074] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1129.782660] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:33:30 (1784414010) [ 1130.419724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1131.049632] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:33:32 (1784414012) [ 1135.586260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.587082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.591608] Lustre: Skipped 2 previous similar messages [ 1135.593866] Lustre: Skipped 1 previous similar message [ 1138.194909] Lustre: server umount lustre-MDT0000 complete [ 1139.383029] LustreError: 93491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414020 with bad export cookie 17073673677890244925 [ 1139.386194] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.387434] LustreError: 93491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1139.508094] Lustre: server umount lustre-MDT0001 complete [ 1151.107441] Lustre: server umount lustre-OST0000 complete [ 1154.713904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.841699] LustreError: 97796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.847352] LustreError: 97796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1154.890308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.183420] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1158.908659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.111245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1160.470564] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1162.775299] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.526673] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.557415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.630712] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1168.635029] LustreError: 97785:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1168.646075] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1168.650103] LustreError: 99877:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1168.653356] LustreError: 99877:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1168.655285] LustreError: 99877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1168.680060] Lustre: server umount lustre-OST0000 complete [ 1168.681281] LustreError: 99877:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1171.409072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1171.488461] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1171.490914] LustreError: 97786:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1171.501207] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1171.505109] LustreError: 100342:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1171.508036] LustreError: 100342:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1171.510421] LustreError: 100342:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1171.533516] Lustre: server umount lustre-OST0000 complete [ 1171.535169] LustreError: 100342:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1174.483673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1174.545853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1174.548738] Lustre: Found index 0 for lustre-OST0000, updating log [ 1174.551867] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1174.571858] Lustre: lustre-OST0000: new disk, initializing [ 1174.573622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1174.576266] Lustre: 100873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1174.657856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.513215] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1177.652131] Lustre: Failing over lustre-OST0000 [ 1177.702183] Lustre: server umount lustre-OST0000 complete [ 1179.616721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1179.618524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.621581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.225381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.228079] Lustre: Skipped 3 previous similar messages [ 1184.919474] Lustre: server umount lustre-MDT0000 complete [ 1186.205209] LustreError: 99897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414067 with bad export cookie 17073673677890247655 [ 1186.208064] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.209339] LustreError: 99897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1186.324706] Lustre: server umount lustre-MDT0001 complete [ 1189.025112] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:34:30 (1784414070) [ 1194.162945] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1197.594391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.720205] LustreError: 103213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.747889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.149235] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1201.975245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.594864] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1204.568703] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1206.756910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.893637] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1211.885607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1211.924158] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1211.967720] Lustre: lustre-OST0001: new disk, initializing [ 1211.969715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1211.972385] Lustre: 105530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1214.031513] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1217.028868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1217.032224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1217.062528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1217.363556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1226.161678] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1227.744957] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.747314] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.750697] Lustre: Skipped 1 previous similar message [ 1227.752343] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1232.353700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1232.355756] Lustre: Skipped 1 previous similar message [ 1232.356541] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.360445] Lustre: Skipped 1 previous similar message [ 1233.623986] Lustre: server umount lustre-MDT0001 complete [ 1235.654307] LustreError: 103209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.589484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.619469] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1236.621149] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1236.635766] Lustre: 103203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1236.853437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1236.856029] Lustre: Skipped 3 previous similar messages [ 1237.989695] LustreError: 107244:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1237.992259] LustreError: 107244:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1237.995050] LustreError: 103205:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1238.019863] LustreError: 107252:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -17 [ 1238.022707] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1238.022707] [ 1238.124364] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1241.176629] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:35:22 (1784414122) [ 1242.092828] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1246.541576] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1251.930583] Lustre: 109136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1259.572385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1260.848455] Lustre: 110268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1308.640676] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.641417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.645448] Lustre: Skipped 3 previous similar messages [ 1308.646850] Lustre: Skipped 4 previous similar messages [ 1308.650681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.956962] Lustre: server umount lustre-MDT0000 complete [ 1313.092677] LustreError: 105529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414194 with bad export cookie 17073673677890248565 [ 1313.093557] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.096109] LustreError: 105529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1313.198586] Lustre: server umount lustre-MDT0001 complete [ 1323.585405] Lustre: server umount lustre-OST0000 complete [ 1334.947729] Lustre: server umount lustre-OST0001 complete [ 1336.904040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1339.911159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1342.603487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1345.480732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1351.221958] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1354.386139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.394105] Lustre: lustre-MDT0000: reset Object Index mappings [ 1354.525166] LustreError: 114204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.545206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.861241] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1358.597978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.605160] Lustre: lustre-MDT0001: reset Object Index mappings [ 1360.177379] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1361.143649] Lustre: 115351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1363.409295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.707402] LustreError: 115703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.713702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1365.633967] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1368.608442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.837912] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1369.838314] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1369.881676] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1370.616594] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1373.751886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1374.940607] Lustre: 117221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1389.046799] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:37:50 (1784414270) [ 1420.257065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1420.260506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.265092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.267177] Lustre: Skipped 1 previous similar message [ 1424.614992] Lustre: server umount lustre-MDT0000 complete [ 1425.798118] LustreError: 114186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414307 with bad export cookie 17073673677890295304 [ 1425.801044] LustreError: 114186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1425.802223] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1426.400994] LustreError: 114201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.407049] LustreError: 114201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1432.185378] Lustre: server umount lustre-MDT0001 complete [ 1439.559218] Lustre: server umount lustre-OST0000 complete [ 1446.995950] Lustre: server umount lustre-OST0001 complete [ 1448.994088] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 1451.914147] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1462.567101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.653317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1462.663806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1462.704964] Lustre: lustre-MDT0000: new disk, initializing [ 1462.730341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1464.195172] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1467.088350] Lustre: server umount lustre-MDT0000 complete [ 1469.943623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.028090] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1470.031715] Lustre: 121476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1470.035411] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1471.332147] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1472.863787] Lustre: server umount lustre-MDT0000 complete [ 1475.476131] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:39:16 (1784414356) [ 1475.973586] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1476.560441] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:39:17 (1784414357) [ 1481.977054] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1485.090871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.223369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.225140] Lustre: Skipped 5 previous similar messages [ 1486.444874] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1489.076467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.123220] Lustre: 123523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1489.148546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1489.150490] Lustre: Skipped 1 previous similar message [ 1489.185888] Lustre: lustre-MDT0001: new disk, initializing [ 1489.234924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1489.241634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1490.532700] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1492.780213] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1495.029574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.135516] Lustre: lustre-OST0000: new disk, initializing [ 1495.137950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1495.141584] Lustre: 125099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1496.377925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1496.381440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1496.420438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1497.170904] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1501.392407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.455642] Lustre: 125968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1503.048691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1503.051744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1503.089601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1503.483205] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1507.950296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1509.178073] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1653.042655] Lustre: 132347:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1653.045471] Lustre: 132347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1653.047954] Lustre: 132347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1653.050591] Lustre: 132347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1653.053210] Lustre: 132347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1653.055445] Lustre: 132347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1725.701941] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:43:26 (1784414606) [ 1737.699331] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.701087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.704846] Lustre: Skipped 7 previous similar messages [ 1737.707066] Lustre: Skipped 9 previous similar messages [ 1743.481809] Lustre: server umount lustre-MDT0000 complete [ 1744.678165] LustreError: 124261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414626 with bad export cookie 17073673677890348140 [ 1744.681090] LustreError: 124261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1744.681415] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.936892] LustreError: 126767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1747.940641] LustreError: 126767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1750.847222] Lustre: server umount lustre-MDT0001 complete [ 1766.368167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.411600] Lustre: server umount lustre-OST0000 complete [ 1773.788912] Lustre: server umount lustre-OST0001 complete [ 1776.002922] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 1779.595948] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1791.520089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.615137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1791.624858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1791.627732] Lustre: Skipped 1 previous similar message [ 1791.672561] Lustre: lustre-MDT0000: new disk, initializing [ 1791.674633] Lustre: Skipped 1 previous similar message [ 1791.697480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.700419] Lustre: Skipped 3 previous similar messages [ 1791.706446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1793.090190] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1797.298916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.340967] Lustre: 135361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.442494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1798.614326] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1802.267093] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1804.143518] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1806.452815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.581205] Lustre: 137328:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1807.933194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1807.937062] Lustre: Skipped 1 previous similar message [ 1807.938910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1807.974402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1808.684255] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1812.485960] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1815.616213] Lustre: lustre-MDD0000: changelog on [ 1816.523818] Lustre: lustre-MDD0001: changelog on [ 2147.425415] Lustre: 135367:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2174.448350] Lustre: lustre-MDD0001: changelog off [ 2175.588112] Lustre: lustre-MDD0000: changelog off [ 2179.555211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.556131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.560516] Lustre: Skipped 4 previous similar messages [ 2179.562516] Lustre: Skipped 10 previous similar messages [ 2183.324426] Lustre: server umount lustre-OST0000 complete [ 2190.775033] Lustre: server umount lustre-MDT0000 complete [ 2191.988041] LustreError: 135352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415073 with bad export cookie 17073673677891115984 [ 2191.991376] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2191.992271] LustreError: 135352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2192.117291] Lustre: server umount lustre-MDT0001 complete [ 2195.678897] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2196.683710] Key type lgssc unregistered [ 2196.826385] LNet: 146092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.828654] LNetError: 146092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.836338] LNet: Removed LNI 192.168.201.117@tcp [ 2197.165138] Key type .llcrypt unregistered [ 2197.166461] Key type ._llcrypt unregistered [ 2203.348731] Key type ._llcrypt registered [ 2203.349704] Key type .llcrypt registered [ 2203.380532] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2209.333107] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2209.628311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.666381] alg: No test for adler32 (adler32-zlib) [ 2210.507943] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2210.580399] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2212.152125] Key type lgssc registered [ 2212.469065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2222.862770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2222.867825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.940145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2223.949882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2223.984252] Lustre: lustre-MDT0000: new disk, initializing [ 2224.005229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.011375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2225.264057] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2229.052709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.091417] Lustre: 149267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2229.110730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2229.112320] Lustre: Skipped 1 previous similar message [ 2229.145509] Lustre: lustre-MDT0001: new disk, initializing [ 2229.175118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.188512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2229.191558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2230.382383] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2233.884573] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.561137] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.571479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2237.667920] Lustre: lustre-OST0000: new disk, initializing [ 2237.669869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2237.672777] Lustre: 151233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2237.702360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.506523] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2243.177559] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2243.574547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2243.578032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2243.612226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2250.520473] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_module obdecho/obdecho [ 2251.149234] LustreError: 152239:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2251.153757] LustreError: 152239:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2251.760594] LustreError: 152335:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 2258.914282] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.915416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.918081] Lustre: Skipped 2 previous similar messages [ 2258.919839] Lustre: Skipped 1 previous similar message [ 2261.076204] Lustre: server umount lustre-MDT0000 complete [ 2262.179191] LustreError: 151232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415143 with bad export cookie 7865130614186249884 [ 2262.182302] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.183142] LustreError: 151232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2262.295824] Lustre: server umount lustre-MDT0001 complete [ 2273.758253] Lustre: server umount lustre-OST0000 complete [ 2280.536060] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2283.394831] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2294.915226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.987711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2294.997676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2295.031221] Lustre: lustre-MDT0000: new disk, initializing [ 2295.048259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.052920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2296.284926] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2300.139305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.178960] Lustre: 156051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2300.207629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2300.208901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2300.209145] Lustre: Skipped 1 previous similar message [ 2300.212045] Lustre: Skipped 1 previous similar message [ 2300.244550] Lustre: lustre-MDT0001: new disk, initializing [ 2300.269588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2300.280678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2300.283410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2301.412529] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2305.058284] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2306.914993] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.137914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.245103] Lustre: lustre-OST0000: new disk, initializing [ 2309.246702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2309.248961] Lustre: 158016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2309.273703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.165276] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2314.842842] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2316.206371] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2316.277846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2316.280632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2316.313423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2317.296752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2318.660031] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2318.732507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2321.378593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2321.379802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.380716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.381089] LustreError: Skipped 1 previous similar message [ 2321.384407] Lustre: Skipped 1 previous similar message [ 2325.523078] Lustre: server umount lustre-OST0000 complete [ 2331.104950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.107397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.111520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.113083] Lustre: Skipped 1 previous similar message [ 2336.737183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.739229] Lustre: Skipped 2 previous similar messages [ 2341.344130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2341.413160] Lustre: server umount lustre-MDT0000 complete [ 2342.710064] LustreError: 156042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415224 with bad export cookie 7865130614186250829 [ 2342.713162] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2342.713981] LustreError: 156042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2342.826658] Lustre: server umount lustre-MDT0001 complete [ 2344.032233] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:53:45 (1784415225) [ 2344.576070] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2345.177499] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:53:46 (1784415226) [ 2347.761216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.909616] LustreError: 160429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.915042] LustreError: 160429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2347.930409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.281927] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2352.172180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.672751] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2356.221073] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2358.167133] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2360.443676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2362.582254] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2365.049488] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2378.208886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.211575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.215925] Lustre: Skipped 1 previous similar message [ 2378.217975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.464660] Lustre: server umount lustre-MDT0000 complete [ 2384.353505] LustreError: 160429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.358830] LustreError: 160429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2384.707109] LustreError: 160411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415266 with bad export cookie 7865130614186251711 [ 2384.711045] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.711407] LustreError: 160411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2385.605457] LustreError: 160425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.472931] LustreError: 160426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.473231] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.483446] Lustre: Skipped 2 previous similar messages [ 2394.592978] LustreError: 161861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.593257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2394.600316] LustreError: 161861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2394.602336] Lustre: Skipped 7 previous similar messages [ 2399.200086] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2399.282497] Lustre: server umount lustre-MDT0001 complete [ 2401.265992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2405.304200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.381158] LustreError: 164289:0:(llog.c:1655:llog_backup()) MGC192.168.201.117@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2405.383715] Lustre: 164289:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.117@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2420.192920] LustreError: 164294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.199258] LustreError: 164294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2436.805198] LustreError: 164296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.815053] LustreError: 164296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 2466.784285] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2466.785974] Lustre: 164289:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2469.856844] LustreError: 164294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.861207] LustreError: 164294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 2507.311285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2507.313304] Lustre: Skipped 2 previous similar messages [ 2508.684836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2509.348086] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2511.509664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.764308] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6d268e9f529f2eee [ 2529.766764] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [ 2531.252181] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2533.662267] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.160871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2534.890316] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2535.909404] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2538.447176] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2543.010735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.011544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.014054] Lustre: Skipped 1 previous similar message [ 2543.015809] Lustre: Skipped 1 previous similar message [ 2544.855083] Lustre: server umount lustre-MDT0000 complete [ 2546.040274] LustreError: 160411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415427 with bad export cookie 7865130614186258158 [ 2546.043346] LustreError: 160411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2546.043487] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.155167] Lustre: server umount lustre-MDT0001 complete [ 2556.628807] Lustre: server umount lustre-OST0000 complete [ 2563.380875] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2566.482501] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2578.431094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.532362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2578.543122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2578.586481] Lustre: lustre-MDT0000: new disk, initializing [ 2578.612518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.615665] Lustre: Skipped 1 previous similar message [ 2578.623095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2579.977223] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2584.286931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.327412] Lustre: 169369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2584.436191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2585.875308] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2589.580047] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2591.404201] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2593.657041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.791716] Lustre: 171338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2595.785908] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2599.484071] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2599.929271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2599.961181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2600.974372] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2601.049447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2602.426542] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2602.498740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2605.029649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.031493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.031882] LustreError: Skipped 1 previous similar message [ 2605.039614] Lustre: Skipped 1 previous similar message [ 2609.300939] Lustre: server umount lustre-OST0000 complete [ 2615.267078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2615.268112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.273912] Lustre: Skipped 5 previous similar messages [ 2616.865133] Lustre: server umount lustre-MDT0000 complete [ 2618.098342] LustreError: 171337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415499 with bad export cookie 7865130614186274986 [ 2618.101323] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.101406] LustreError: 171337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2618.228762] Lustre: server umount lustre-MDT0001 complete [ 2620.977887] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:58:22 (1784415502) [ 2623.104727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.260991] LustreError: 173748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.266309] LustreError: 173748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 2624.610252] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2627.375194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.809444] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2631.088127] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2632.855576] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2634.964881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.949968] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2639.368785] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2653.537487] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:58:54 (1784415534) [ 2658.785610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.786709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.788148] LustreError: Skipped 1 previous similar message [ 2658.792483] Lustre: Skipped 4 previous similar messages [ 2660.618312] Lustre: server umount lustre-OST0000 complete [ 2676.192142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2676.252285] Lustre: server umount lustre-MDT0000 complete [ 2677.456195] LustreError: 174483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415559 with bad export cookie 7865130614186275868 [ 2677.459475] LustreError: 174483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2677.459691] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.574101] Lustre: server umount lustre-MDT0001 complete [ 2680.926156] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2681.934334] Key type lgssc unregistered [ 2682.077520] LNet: 177495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.079689] LNetError: 177495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.089397] LNet: Removed LNI 192.168.201.117@tcp [ 2682.410489] Key type .llcrypt unregistered [ 2682.412152] Key type ._llcrypt unregistered [ 2689.782490] Key type ._llcrypt registered [ 2689.783497] Key type .llcrypt registered [ 2689.826646] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2690.254308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.300090] alg: No test for adler32 (adler32-zlib) [ 2691.166791] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2691.271944] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2692.856209] Key type lgssc registered [ 2693.232130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2695.630327] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2695.635777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.072337] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2716.192286] LustreError: 178714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 2721.312466] LustreError: 178714:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2721.317829] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2721.319312] LustreError: MGC192.168.201.117@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2721.322070] LustreError: 178714:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2721.324216] LustreError: 178714:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 2721.326097] LustreError: 178714:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2721.348331] Lustre: server umount lustre-OST0000 complete [ 2721.349900] LustreError: 178714:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2724.124358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.263301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.540459] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2729.288628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.432649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.713786] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2733.069353] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2734.838900] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2737.170278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.243644] LustreError: 181024:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2737.249065] Lustre: 181024:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.117@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2737.295960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.150364] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2741.489448] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2744.803324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.804571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.807120] Lustre: Skipped 2 previous similar messages [ 2744.807354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2744.809011] Lustre: Skipped 2 previous similar messages [ 2748.307809] Lustre: server umount lustre-MDT0000 complete [ 2749.452236] LustreError: 179194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415630 with bad export cookie 5764688224282697249 [ 2749.455524] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.456687] LustreError: 179194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2749.559338] Lustre: server umount lustre-MDT0001 complete [ 2760.989763] Lustre: server umount lustre-OST0000 complete [ 2763.933380] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2766.946163] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2779.786351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.875242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2779.883912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2779.919459] Lustre: lustre-MDT0000: new disk, initializing [ 2779.940059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.945571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2781.348343] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2785.420425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.462942] Lustre: 184552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2785.491793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2785.493403] Lustre: Skipped 1 previous similar message [ 2785.529220] Lustre: lustre-MDT0001: new disk, initializing [ 2785.562514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.574583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2785.579500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2786.984571] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2790.754262] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2792.495986] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2794.601946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.747862] Lustre: lustre-OST0000: new disk, initializing [ 2794.749413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2794.751691] Lustre: 186519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2794.783269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.966201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2794.970589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2795.013861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2796.649415] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2800.151105] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2801.599964] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2801.679594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2803.081775] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2803.152029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2805.217510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2805.219341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.219751] LustreError: Skipped 1 previous similar message [ 2805.223581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.226699] Lustre: Skipped 1 previous similar message [ 2805.839507] Lustre: server umount lustre-OST0000 complete [ 2810.337550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.338432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.342958] Lustre: Skipped 2 previous similar messages [ 2813.148667] Lustre: server umount lustre-MDT0000 complete [ 2814.353297] LustreError: 185344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415695 with bad export cookie 5764688224282697935 [ 2814.356732] LustreError: 185344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2814.357037] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.458088] Lustre: server umount lustre-MDT0001 complete [ 2817.380286] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:01:38 (1784415698) [ 2823.127026] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2824.110540] Key type lgssc unregistered [ 2824.237423] LNet: 189455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.239526] LNetError: 189455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2824.249306] LNet: Removed LNI 192.168.201.117@tcp [ 2824.550889] Key type .llcrypt unregistered [ 2824.551910] Key type ._llcrypt unregistered [ 2831.218433] Key type ._llcrypt registered [ 2831.219429] Key type .llcrypt registered [ 2831.256255] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2831.673831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2831.680268] alg: No test for adler32 (adler32-zlib) [ 2832.530467] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2832.626287] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2834.208111] Key type lgssc registered [ 2834.556508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2836.950615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2836.955813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.096473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.403218] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2842.144432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.273638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.603422] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2845.983048] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2847.728759] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2850.114049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.210260] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2850.213136] LustreError: 192510:0:(llog.c:1696:llog_backup()) MGC192.168.201.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2850.219632] Lustre: 192510:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.117@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2850.267529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.038781] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2854.221380] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2854.865596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.866978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.869597] Lustre: Skipped 2 previous similar messages [ 2854.872230] Lustre: Skipped 1 previous similar message [ 2860.001649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.003409] Lustre: Skipped 3 previous similar messages [ 2863.009425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.012435] Lustre: Skipped 2 previous similar messages [ 2868.193653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.195607] Lustre: Skipped 2 previous similar messages [ 2869.216106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2869.274746] Lustre: server umount lustre-MDT0000 complete [ 2870.406306] LustreError: 190632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415751 with bad export cookie 13734778190034654992 [ 2870.408372] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.411078] LustreError: 190632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2870.526572] Lustre: server umount lustre-MDT0001 complete [ 2882.006356] Lustre: server umount lustre-OST0000 complete [ 2884.821472] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2887.811486] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2899.156047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.232994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2899.241907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2899.276427] Lustre: lustre-MDT0000: new disk, initializing [ 2899.296060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.301071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2900.653696] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2904.693061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.738250] Lustre: 196026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2904.758065] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2904.759148] Lustre: Skipped 1 previous similar message [ 2904.760240] LustreError: 196809:0:(llog.c:1696:llog_backup()) MGC192.168.201.117@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2904.762495] LustreError: 196809:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 2904.764342] Lustre: 196809:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.117@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2904.770866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2904.772263] Lustre: Skipped 1 previous similar message [ 2904.805672] Lustre: lustre-MDT0001: new disk, initializing [ 2904.835735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.846781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2904.849735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2906.097360] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2909.758046] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2911.506848] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2913.685584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.810812] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2913.812058] Lustre: Skipped 1 previous similar message [ 2913.813436] LustreError: 197972:0:(llog.c:1696:llog_backup()) MGC192.168.201.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2913.816521] LustreError: 197972:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 2913.818419] Lustre: 197972:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.117@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2913.821367] Lustre: 197972:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2913.829585] Lustre: lustre-OST0000: new disk, initializing [ 2913.831172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2913.833243] Lustre: 197996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2913.867606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.710615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2915.714156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2915.759096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2915.873668] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2919.557178] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2920.939820] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2921.017741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2922.629653] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2922.750508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2926.053624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2926.055928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.058367] LustreError: Skipped 1 previous similar message [ 2926.064436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2926.068665] Lustre: Skipped 1 previous similar message [ 2929.614726] Lustre: server umount lustre-OST0000 complete [ 2931.169549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.173509] Lustre: Skipped 2 previous similar messages [ 2936.289658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2936.292223] Lustre: Skipped 3 previous similar messages [ 2936.982924] Lustre: server umount lustre-MDT0000 complete [ 2938.182953] LustreError: 196823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415819 with bad export cookie 13734778190034655692 [ 2938.185795] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2938.185848] LustreError: 196823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2938.296342] Lustre: server umount lustre-MDT0001 complete [ 2941.055585] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:03:42 (1784415822) [ 2946.643422] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2950.192384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.321734] LustreError: 201116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.326430] LustreError: 201116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2950.332902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.751043] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2954.620523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.127398] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2957.178453] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2959.414746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2961.320948] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2964.080305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.153096] Lustre: lustre-OST0001: new disk, initializing [ 2964.154581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2964.156807] Lustre: 203422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2966.192381] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2971.137684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2971.141758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2971.167748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2971.443883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2976.954582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2977.522825] Lustre: DEBUG MARKER: ADD OST3 [ 2979.004351] loop: module loaded [ 2979.024059] loop0: detected capacity change from 0 to 204800000 [ 2979.097902] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2979.103773] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2979.109937] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2981.092234] loop0: detected capacity change from 0 to 204800000 [ 2983.010109] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.089590] Lustre: lustre-OST0003: new disk, initializing [ 2983.091533] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 2983.094348] Lustre: 205152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2983.140827] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 2983.143051] Lustre: Skipped 3 previous similar messages [ 2984.075801] Lustre: DEBUG MARKER: STOP OST3 [ 2984.401453] Lustre: *** cfs_fail_loc=2109, val=2*** [ 2984.456976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 2984.460139] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 2984.471310] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 2984.833606] Lustre: Failing over lustre-OST0003 [ 2984.885483] Lustre: server umount lustre-OST0003 complete [ 2985.406463] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2986.465186] LustreError: 205415:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 2989.229135] Lustre: DEBUG MARKER: START OST3 again [ 2989.542532] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.544191] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.545349] LustreError: 204000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.548430] Lustre: Skipped 1 previous similar message [ 2989.556620] LustreError: 204000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2991.485758] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2991.572303] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 2993.033093] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2993.473061] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2993.577442] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2993.577458] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3006.982824] Lustre: Failing over lustre-OST0003 [ 3007.025184] Lustre: server umount lustre-OST0003 complete [ 3008.993181] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3008.993209] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.995144] LustreError: Skipped 1 previous similar message [ 3008.998750] LustreError: 202600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.004751] LustreError: 202600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3012.070165] LustreError: 202602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.076764] LustreError: 202602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3017.187850] LustreError: 202602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.194366] LustreError: 202602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3021.281166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3021.284276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.289428] Lustre: Skipped 1 previous similar message [ 3021.291655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.638816] Lustre: server umount lustre-MDT0000 complete [ 3026.862499] LustreError: 201097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415908 with bad export cookie 13734778190034656560 [ 3026.868090] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.425810] LustreError: 202786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.434197] LustreError: 202786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3041.248093] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3041.345310] Lustre: server umount lustre-MDT0001 complete [ 3042.743871] Lustre: server umount lustre-OST0000 complete [ 3044.041799] Lustre: server umount lustre-OST0001 complete [ 3046.359577] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3049.094972] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3062.244256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.324220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3062.333544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3062.372082] Lustre: lustre-MDT0000: new disk, initializing [ 3062.392125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.395428] Lustre: Skipped 1 previous similar message [ 3062.401634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3063.929820] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3068.110914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.150166] Lustre: 210563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3068.252385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3069.547466] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3073.207182] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3075.463615] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3078.421177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3078.567994] Lustre: lustre-OST0000: new disk, initializing [ 3078.571470] Lustre: Skipped 1 previous similar message [ 3078.575948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3078.582120] Lustre: Skipped 2 previous similar messages [ 3078.588665] Lustre: 212530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3079.752549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3079.759512] Lustre: Skipped 1 previous similar message [ 3079.763507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3079.807625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3080.924926] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3084.888465] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3086.742256] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3086.815379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3088.249085] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3088.317816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3090.402568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3090.403550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.404700] LustreError: Skipped 1 previous similar message [ 3090.405287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3090.410877] Lustre: Skipped 6 previous similar messages [ 3090.412456] Lustre: Skipped 10 previous similar messages [ 3095.212472] Lustre: server umount lustre-OST0000 complete [ 3099.105511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3102.711354] Lustre: server umount lustre-MDT0000 complete [ 3104.312755] LustreError: 210555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415985 with bad export cookie 13734778190034658842 [ 3104.315808] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.320077] LustreError: 210555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3104.450723] Lustre: server umount lustre-MDT0001 complete [ 3105.905839] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:06:26 (1784415986) [ 3112.645973] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3115.986390] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3127.366025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.453463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3127.463717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3127.499072] Lustre: lustre-MDT0000: new disk, initializing [ 3127.522809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3128.889577] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3133.045371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.098883] Lustre: 217133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3133.196346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3134.512385] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3138.123215] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3139.951065] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3142.198148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.317989] Lustre: 219100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3144.331277] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3144.422981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3148.088688] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3149.474758] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3149.551949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3150.819609] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3150.893779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3154.914632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.915737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.917136] LustreError: Skipped 1 previous similar message [ 3154.921457] Lustre: Skipped 2 previous similar messages [ 3154.922094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.926622] Lustre: Skipped 5 previous similar messages [ 3157.587859] Lustre: server umount lustre-OST0000 complete [ 3173.344138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3173.408509] Lustre: server umount lustre-MDT0000 complete [ 3174.939187] LustreError: 217125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416056 with bad export cookie 13734778190034659724 [ 3174.943067] LustreError: 217125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3174.943805] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.070726] Lustre: server umount lustre-MDT0001 complete [ 3178.174375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.342500] LustreError: 221312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.352065] LustreError: 221312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3179.773442] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3182.616389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.116584] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3186.706340] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3188.626332] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3191.362216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.508216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.510310] Lustre: Skipped 7 previous similar messages [ 3193.588202] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3196.914740] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3213.286501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3213.290925] LustreError: Skipped 2 previous similar messages [ 3215.246766] Lustre: server umount lustre-OST0000 complete [ 3222.703313] Lustre: server umount lustre-MDT0000 complete [ 3223.521734] LustreError: 222039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.538585] LustreError: 222039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3224.524169] LustreError: 221294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416106 with bad export cookie 13734778190034660606 [ 3224.530819] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.532074] LustreError: 221294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3224.682685] Lustre: server umount lustre-MDT0001 complete [ 3331.019702] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3332.280453] Key type lgssc unregistered [ 3332.401427] LNet: 224997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3332.404450] LNetError: 224997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.413527] LNet: Removed LNI 192.168.201.117@tcp [ 3332.881156] Key type .llcrypt unregistered [ 3332.883190] Key type ._llcrypt unregistered [ 3342.425972] Key type ._llcrypt registered [ 3342.427088] Key type .llcrypt registered [ 3342.465119] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3349.307363] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3349.745106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.768881] alg: No test for adler32 (adler32-zlib) [ 3350.620760] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3350.708026] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3352.288160] Key type lgssc registered [ 3352.834905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3367.069436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.074212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.167559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3368.177764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3368.216713] Lustre: lustre-MDT0000: new disk, initializing [ 3368.238409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.246299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3369.725490] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3374.625270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.702972] Lustre: 228323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3374.734213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3374.736404] Lustre: Skipped 1 previous similar message [ 3374.771592] Lustre: lustre-MDT0001: new disk, initializing [ 3374.802722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3374.814067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3374.819585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3376.347198] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3380.589454] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3382.786879] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3385.787228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3385.939535] Lustre: lustre-OST0000: new disk, initializing [ 3385.942795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3385.947482] Lustre: 230292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3385.995280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.208358] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3389.509572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3389.515245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3389.566029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3392.658790] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3394.440240] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3394.548408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3396.152034] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3396.261161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3400.163144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3400.165219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.167882] LustreError: Skipped 1 previous similar message [ 3400.176979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.182067] Lustre: Skipped 1 previous similar message [ 3403.156218] Lustre: server umount lustre-OST0000 complete [ 3405.281974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.282888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.285948] Lustre: Skipped 2 previous similar messages [ 3410.401053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.402974] Lustre: Skipped 2 previous similar messages [ 3415.522071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.526654] Lustre: Skipped 1 previous similar message [ 3419.104149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3419.187156] Lustre: server umount lustre-MDT0000 complete [ 3420.764784] LustreError: 228315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416302 with bad export cookie 5048369497120412637 [ 3420.769579] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.770353] LustreError: 228315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3420.924278] Lustre: server umount lustre-MDT0001 complete [ 3427.149371] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3431.508619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.683499] LustreError: 233222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.690819] LustreError: 233222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3431.702460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.254320] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3436.767127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.953982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3438.513518] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3439.936292] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3442.532784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.810971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3447.844148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.923295] Lustre: lustre-OST0001: new disk, initializing [ 3447.924855] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3447.927179] Lustre: 235527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3447.964996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.967394] Lustre: Skipped 1 previous similar message [ 3450.147036] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3450.707977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3450.711340] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3450.758254] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3455.636342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3459.308095] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3463.069964] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:12:24 (1784416344) [ 3467.234090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3467.238136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.239979] Lustre: Skipped 3 previous similar messages [ 3467.241557] Lustre: Skipped 2 previous similar messages [ 3469.850611] Lustre: server umount lustre-MDT0000 complete [ 3470.964341] LustreError: 236295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416352 with bad export cookie 5048369497120413519 [ 3470.966916] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3470.969109] LustreError: 236295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3471.087610] Lustre: server umount lustre-MDT0001 complete [ 3481.534523] Lustre: server umount lustre-OST0000 complete [ 3493.165485] Lustre: server umount lustre-OST0001 complete [ 3495.481889] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3499.406885] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3511.705176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.787711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3511.796442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3511.830806] Lustre: lustre-MDT0000: new disk, initializing [ 3511.851779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.857457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3513.126922] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3517.120057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.162857] Lustre: 239464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3517.275135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3518.551289] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3522.012463] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3523.656483] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3526.595697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.704664] Lustre: lustre-OST0000: new disk, initializing [ 3526.707684] Lustre: Skipped 1 previous similar message [ 3526.709370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3526.713021] Lustre: Skipped 2 previous similar messages [ 3526.717999] Lustre: 241429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3528.321834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3528.325749] Lustre: Skipped 1 previous similar message [ 3528.328822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3528.362355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3528.912571] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3533.669794] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3535.569419] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3535.680966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3537.400180] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3537.485521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3538.917652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.917684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3538.918994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3538.918999] Lustre: Skipped 1 previous similar message [ 3538.923063] Lustre: Skipped 1 previous similar message [ 3538.935771] LustreError: Skipped 1 previous similar message [ 3544.268443] Lustre: server umount lustre-OST0000 complete [ 3548.128744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3548.131538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.707275] Lustre: server umount lustre-MDT0000 complete [ 3552.812082] LustreError: 239455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416434 with bad export cookie 5048369497120415612 [ 3552.815142] LustreError: 239455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3552.816144] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3552.921696] Lustre: server umount lustre-MDT0001 complete [ 3555.481480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.626147] LustreError: 243643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.647040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.648714] Lustre: Skipped 2 previous similar messages [ 3556.834879] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3559.740099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.065340] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3563.177537] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3564.784659] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.790051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3568.563314] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3570.600648] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3713.506536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3713.510574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.516062] Lustre: Skipped 1 previous similar message [ 3713.518691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.521710] Lustre: Skipped 6 previous similar messages [ 3727.328127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3727.388091] Lustre: server umount lustre-OST0000 complete [ 3729.891128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.899388] Lustre: Skipped 2 previous similar messages [ 3743.200164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3743.282918] Lustre: server umount lustre-MDT0000 complete [ 3744.885969] LustreError: 243625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416626 with bad export cookie 5048369497120416494 [ 3744.891102] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.893327] LustreError: 243625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3745.034795] Lustre: server umount lustre-MDT0001 complete [ 3834.410841] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3835.332873] Key type lgssc unregistered [ 3835.458413] LNet: 247248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3835.460569] LNetError: 247248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3835.469329] LNet: Removed LNI 192.168.201.117@tcp [ 3835.780171] Key type .llcrypt unregistered [ 3835.781302] Key type ._llcrypt unregistered [ 3845.263748] Key type ._llcrypt registered [ 3845.265139] Key type .llcrypt registered [ 3845.315569] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3851.904079] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3852.199462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3852.238478] alg: No test for adler32 (adler32-zlib) [ 3853.106218] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3853.201180] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3854.784215] Key type lgssc registered [ 3855.294113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3873.654605] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3877.670282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3877.680462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.791906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3878.806214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3878.847911] Lustre: lustre-MDT0000: new disk, initializing [ 3878.868985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.877086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3880.481372] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3885.902751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.974785] Lustre: 251287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3886.023476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3886.027502] Lustre: Skipped 1 previous similar message [ 3886.073115] Lustre: lustre-MDT0001: new disk, initializing [ 3886.110902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3886.127109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3886.131881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3888.017091] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3890.736837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3893.590181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3893.758191] Lustre: lustre-OST0000: new disk, initializing [ 3893.760936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3893.764859] Lustre: 252922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3893.816315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.507440] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3896.853228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3896.858793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3896.934851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3901.082262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.191491] Lustre: lustre-OST0001: new disk, initializing [ 3901.194322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3901.198979] Lustre: 253795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3901.254056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3903.284316] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3907.081363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3907.085033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3907.126568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3908.463536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3913.011879] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3916.769028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3916.770936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.774449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.285630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.287607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.289200] Lustre: Skipped 2 previous similar messages [ 3917.291450] Lustre: Skipped 2 previous similar messages [ 3922.028060] Lustre: server umount lustre-MDT0000 complete [ 3923.227493] LustreError: 251282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416804 with bad export cookie 17597205821210605235 [ 3923.232307] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.349894] Lustre: server umount lustre-MDT0001 complete [ 3934.732121] Lustre: server umount lustre-OST0000 complete [ 3946.438241] Lustre: server umount lustre-OST0001 complete [ 3952.051894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3956.273628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3959.555085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3963.682090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3971.246942] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3975.330025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.338744] Lustre: lustre-MDT0000: reset Object Index mappings [ 3975.483393] LustreError: 258793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.490306] LustreError: 258793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3975.498843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.861339] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3980.168350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.178424] Lustre: lustre-MDT0001: reset Object Index mappings [ 3981.680971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3982.783857] Lustre: 259935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3985.802090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.997669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.004754] Lustre: Skipped 1 previous similar message [ 3987.049846] LustreError: 260290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.059617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3988.536624] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3990.119320] LustreError: 260289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.129916] LustreError: 260289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3992.225669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.428314] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3997.675996] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 3997.853115] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3999.429416] Lustre: 261802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4002.791245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4002.793766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4002.799935] Lustre: Skipped 2 previous similar messages [ 4002.803628] Lustre: Skipped 1 previous similar message [ 4006.957696] Lustre: server umount lustre-MDT0000 complete [ 4007.906240] LustreError: 260238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.910764] LustreError: 260238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4008.271911] LustreError: 258776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416889 with bad export cookie 17597205821210612011 [ 4008.277516] LustreError: 258776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4008.279228] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.406506] Lustre: server umount lustre-MDT0001 complete [ 4020.244390] Lustre: server umount lustre-OST0000 complete [ 4032.190507] Lustre: server umount lustre-OST0001 complete [ 4040.621065] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:22:01 (1784416921) [ 4048.583245] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4053.214632] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4069.589073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.708251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4069.725844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4069.783458] Lustre: lustre-MDT0000: new disk, initializing [ 4069.812639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.816373] Lustre: Skipped 1 previous similar message [ 4069.824587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4071.681574] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4077.116210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.189775] Lustre: 266052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4077.354712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4079.251775] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4083.611386] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4086.122143] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4089.167851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4089.314497] Lustre: lustre-OST0000: new disk, initializing [ 4089.317626] Lustre: Skipped 1 previous similar message [ 4089.320877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4089.324466] Lustre: Skipped 2 previous similar messages [ 4089.327900] Lustre: 268019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4090.700549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4090.704405] Lustre: Skipped 1 previous similar message [ 4090.707124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4090.751391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4091.881836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4096.541303] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4098.358194] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4098.459420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4100.282759] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4100.397969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4106.211260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4106.211480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.213162] LustreError: Skipped 1 previous similar message [ 4106.218592] Lustre: Skipped 1 previous similar message [ 4106.220197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.222727] Lustre: Skipped 3 previous similar messages [ 4107.426722] Lustre: server umount lustre-OST0000 complete [ 4111.330964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.332322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4111.337272] Lustre: Skipped 1 previous similar message [ 4111.343693] Lustre: Skipped 1 previous similar message [ 4114.809876] Lustre: server umount lustre-MDT0000 complete [ 4116.449109] LustreError: 266863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.458746] LustreError: 266863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4116.519298] LustreError: 266847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416998 with bad export cookie 17597205821210614083 [ 4116.523630] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.525586] LustreError: 266847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4116.656775] Lustre: server umount lustre-MDT0001 complete [ 4123.520545] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4126.982844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.109564] LustreError: 270947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.122426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.124116] Lustre: Skipped 2 previous similar messages [ 4128.408176] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4132.225564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.320850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4133.916278] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4135.171468] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4137.671754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4139.825393] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4143.088585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.154595] Lustre: lustre-OST0001: new disk, initializing [ 4143.156337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4143.159094] Lustre: 273251:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4145.092499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4145.095667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4145.128131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4145.338840] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4150.221265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4151.550879] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4152.303549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4153.392740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4153.398505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4157.411072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.413754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.416061] Lustre: Skipped 3 previous similar messages [ 4157.417437] Lustre: Skipped 2 previous similar messages [ 4161.659329] Lustre: server umount lustre-MDT0000 complete [ 4162.530179] LustreError: 270948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.539032] LustreError: 270948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4163.311846] LustreError: 270928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417044 with bad export cookie 17597205821210614965 [ 4163.318824] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4163.319149] LustreError: 270928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4163.478279] Lustre: server umount lustre-MDT0001 complete [ 4175.900108] Lustre: server umount lustre-OST0000 complete [ 4187.957028] Lustre: server umount lustre-OST0001 complete [ 4196.227397] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4200.727619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.931146] LustreError: 276309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.959149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.961758] Lustre: Skipped 3 previous similar messages [ 4202.848426] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4207.197250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.835570] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4210.297592] Lustre: 277452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4213.656402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4216.422336] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4220.521360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.328949] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4225.771715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:4 to 0x280000403:65) [ 4225.788224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4232.305462] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4234.056441] Lustre: 279330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4236.359141] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4236.383629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000403 to 0x280000404 [ 4240.416812] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:25:21 (1784417121) [ 4253.665568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4253.671831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.683573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.687621] Lustre: Skipped 1 previous similar message [ 4258.496174] Lustre: server umount lustre-MDT0000 complete [ 4260.333236] LustreError: 276290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417141 with bad export cookie 17597205821210621272 [ 4260.339077] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.341288] LustreError: 276290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4265.890215] LustreError: 276304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.901607] LustreError: 276304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4266.599295] Lustre: server umount lustre-MDT0001 complete [ 4282.336177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4282.388894] Lustre: server umount lustre-OST0000 complete [ 4298.208274] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4298.307729] Lustre: server umount lustre-OST0001 complete [ 4300.929938] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4304.927256] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4326.319890] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4330.800934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.950187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4330.967559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4331.014193] Lustre: lustre-MDT0000: new disk, initializing [ 4331.047111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.049939] Lustre: Skipped 3 previous similar messages [ 4331.057891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4332.464575] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4338.089195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4338.159587] Lustre: 283561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4338.313411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4340.283620] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4346.605589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.774191] Lustre: 285198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4346.776939] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4346.776952] Lustre: Skipped 8 previous similar messages [ 4349.380947] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4352.029532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4354.915825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4354.999743] Lustre: 286066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4356.747046] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4357.181786] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4362.560625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4363.980446] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4363.981977] Lustre: Skipped 1 previous similar message [ 4368.470761] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:27:29 (1784417249) [ 4372.456312] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.463980] Lustre: Skipped 7 previous similar messages [ 4375.586175] Lustre: server umount lustre-MDT0000 complete [ 4377.033923] LustreError: 283555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417258 with bad export cookie 17597205821210629154 [ 4377.037808] LustreError: 283555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4377.038525] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4377.177466] Lustre: server umount lustre-MDT0001 complete [ 4389.127977] Lustre: server umount lustre-OST0000 complete [ 4400.854726] Lustre: server umount lustre-OST0001 complete [ 4404.576401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.801408] LustreError: 288308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.495926] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4410.138919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.942087] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4415.180503] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4417.550149] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4420.156216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4422.840844] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4425.555397] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4441.814513] loop1: detected capacity change from 0 to 204800000 [ 4441.898988] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4441.902266] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4441.905406] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4443.498840] loop1: detected capacity change from 0 to 204800000 [ 4445.272431] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.303288] LustreError: 288297:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21963: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4445.308894] LustreError: 288297:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21963 [ 4445.311545] LustreError: 288297:0:(mgs_handler.c:517:mgs_target_reg()) fs21963-MDT0001: Error checking target index: rc = -13 [ 4445.316017] LustreError: fs21963-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4445.319858] LustreError: 292044:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4445.322775] LustreError: 292044:0:(tgt_mount.c:2129:server_put_super()) no obd fs21963-MDT0001 [ 4445.325638] LustreError: 292044:0:(tgt_mount.c:116:server_deregister_mount()) fs21963-MDT0001 not registered [ 4445.338735] Lustre: server umount fs21963-MDT0001 complete [ 4445.340201] LustreError: 292044:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4448.207046] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4448.233074] Lustre: Setting parameter fs21963-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21963-MDT0001 [ 4448.248139] Lustre: srv-fs21963-MDT0001: No data found on store. Initialize space. [ 4448.249688] Lustre: Skipped 4 previous similar messages [ 4448.284601] Lustre: fs21963-MDT0001: new disk, initializing [ 4448.285833] Lustre: Skipped 3 previous similar messages [ 4448.316458] Lustre: srv-fs21963-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4448.318489] Lustre: 292509:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21963-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4448.320708] LustreError: 292509:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21963-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4449.521103] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4452.873365] Lustre: server umount fs21963-MDT0001 complete [ 4457.918013] loop1: detected capacity change from 0 to 204800000 [ 4460.371973] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.492653] Lustre: 294074:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21963-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4460.497095] LustreError: 294074:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21963-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4462.056299] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4463.874729] Lustre: server umount fs21963-MDT0001 complete [ 4470.755233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.762507] Lustre: Skipped 3 previous similar messages [ 4474.099190] Lustre: server umount lustre-MDT0000 complete [ 4477.783334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.836962] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.843719] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4483.047849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4483.160391] loop1: detected capacity change from 0 to 204800000 [ 4485.697953] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.879527] Lustre: 296518:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21963-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4485.885187] LustreError: 296518:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21963-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4487.644289] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4489.758825] Lustre: server umount fs21963-MDT0001 complete [ 4501.473426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4501.479135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4501.482775] Lustre: Skipped 8 previous similar messages [ 4513.248109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4513.300233] Lustre: server umount lustre-OST0000 complete [ 4518.882951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4518.885053] LustreError: Skipped 1 previous similar message [ 4520.601100] Lustre: server umount lustre-MDT0000 complete [ 4521.813468] LustreError: 288288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417403 with bad export cookie 17597205821210631828 [ 4521.819566] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.950228] Lustre: server umount lustre-MDT0001 complete [ 4525.156923] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4526.064019] Key type lgssc unregistered [ 4526.182358] LNet: 298859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4526.184655] LNetError: 298859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4526.194277] LNet: Removed LNI 192.168.201.117@tcp [ 4526.479097] Key type .llcrypt unregistered [ 4526.480091] Key type ._llcrypt unregistered [ 4533.466946] Key type ._llcrypt registered [ 4533.467896] Key type .llcrypt registered [ 4533.500072] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4533.853176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4533.869912] alg: No test for adler32 (adler32-zlib) [ 4534.709341] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4534.788689] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4536.360112] Key type lgssc registered [ 4536.657832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4538.952879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4538.959593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.092088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.307924] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4543.985518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.113992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4545.325913] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4547.476492] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4549.069214] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4551.082310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.192449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.919458] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4554.276319] LustreError: 301932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.283273] LustreError: 301932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4555.221232] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4559.335814] LustreError: 301932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.460623] LustreError: 301935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.599738] LustreError: 301932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.605559] LustreError: 301932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4573.106714] loop1: detected capacity change from 0 to 204800000 [ 4573.212964] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4573.216617] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4573.219937] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4575.626716] loop1: detected capacity change from 0 to 204800000 [ 4575.719840] LustreError: 301935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.724475] LustreError: 301935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4578.053245] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4578.128564] Lustre: fs23342-OST0000: new disk, initializing [ 4578.131664] Lustre: srv-fs23342-OST0000: No data found on store. Initialize space. [ 4578.136312] Lustre: 303858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4578.177950] Lustre: fs23342-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4580.822180] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4585.279469] Lustre: server umount fs23342-OST0000 complete [ 4585.963064] LustreError: 301935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.973842] LustreError: 301935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4591.458699] loop1: detected capacity change from 0 to 204800000 [ 4594.175702] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.292264] Lustre: fs23342-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.114838] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4601.212873] loop2: detected capacity change from 0 to 204800000 [ 4601.312791] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4601.317895] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4601.327459] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4603.819227] loop2: detected capacity change from 0 to 204800000 [ 4606.447902] LustreError: 302478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.457409] LustreError: 302478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4606.551987] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.599530] LustreError: fs23342-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4606.603854] LustreError: fs23342-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4606.608365] LustreError: 306895:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4606.611867] LustreError: 306895:0:(tgt_mount.c:2129:server_put_super()) no obd fs23342-OST0001 [ 4606.616377] LustreError: 306895:0:(tgt_mount.c:116:server_deregister_mount()) fs23342-OST0001 not registered [ 4606.632359] Lustre: server umount fs23342-OST0001 complete [ 4606.634804] LustreError: 306895:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4611.627410] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4611.697384] Lustre: fs23342-OST0001: new disk, initializing [ 4611.701305] Lustre: srv-fs23342-OST0001: No data found on store. Initialize space. [ 4611.707219] Lustre: 307446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4611.750290] Lustre: fs23342-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4614.326142] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4618.605161] Lustre: server umount fs23342-OST0000 complete [ 4621.845747] Lustre: server umount fs23342-OST0001 complete [ 4626.906498] loop1: detected capacity change from 0 to 204800000 [ 4629.461374] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4629.536472] Lustre: fs23342-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.064275] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4635.247218] loop2: detected capacity change from 0 to 204800000 [ 4637.730394] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.243399] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4642.275540] LustreError: 301934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.281220] LustreError: 301934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 4642.282180] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 4644.440332] Lustre: server umount fs23342-OST0000 complete [ 4648.681236] Lustre: server umount fs23342-OST0001 complete [ 4652.517131] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 4656.610451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4656.618355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4656.631583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4657.636479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.638766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4657.648584] Lustre: Skipped 1 previous similar message [ 4659.734321] Lustre: server umount lustre-OST0000 complete [ 4661.729532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4661.734174] LustreError: Skipped 1 previous similar message [ 4661.736952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.745053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.376186] Lustre: server umount lustre-MDT0000 complete [ 4669.131556] LustreError: 300088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417550 with bad export cookie 9315469219210593422 [ 4669.136798] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.139391] LustreError: 300088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4669.286847] Lustre: server umount lustre-MDT0001 complete [ 4673.529345] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4674.793927] Key type lgssc unregistered [ 4674.957568] LNet: 313359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4674.962383] LNetError: 313359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4674.978547] LNet: Removed LNI 192.168.201.117@tcp [ 4675.438176] Key type .llcrypt unregistered [ 4675.440387] Key type ._llcrypt unregistered [ 4688.510300] Key type ._llcrypt registered [ 4688.511303] Key type .llcrypt registered [ 4688.545204] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4695.815896] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4696.152971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4696.186053] alg: No test for adler32 (adler32-zlib) [ 4697.057897] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4697.149968] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4698.720105] Key type lgssc registered [ 4699.101838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4713.138180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4713.152700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.245252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4714.254679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4714.290162] Lustre: lustre-MDT0000: new disk, initializing [ 4714.311635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4714.317572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4715.681399] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4719.967283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.006466] Lustre: 317189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4720.033878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4720.035493] Lustre: Skipped 1 previous similar message [ 4720.073715] Lustre: lustre-MDT0001: new disk, initializing [ 4720.110513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4720.121420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4720.124821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4721.797253] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4725.732207] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4728.125927] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4730.451294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4730.610744] Lustre: lustre-OST0000: new disk, initializing [ 4730.613756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4730.618368] Lustre: 319156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4730.665196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.127340] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4737.352820] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4738.372472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4738.376849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4738.415071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4743.673336] Lustre: Mounted lustre-client [ 4744.095320] Lustre: Unmounted lustre-client [ 4748.771044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4748.775177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.777604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4748.777851] LustreError: Skipped 1 previous similar message [ 4748.785150] Lustre: Skipped 1 previous similar message [ 4750.002344] Lustre: server umount lustre-OST0000 complete [ 4753.889645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.890726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4753.892845] Lustre: Skipped 1 previous similar message [ 4753.897264] Lustre: Skipped 2 previous similar messages [ 4757.687667] Lustre: server umount lustre-MDT0000 complete [ 4759.117413] LustreError: 317182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417640 with bad export cookie 9052975687979110121 [ 4759.119679] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4759.121862] LustreError: 317182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4759.233233] Lustre: server umount lustre-MDT0001 complete [ 4763.233214] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4764.073349] Key type lgssc unregistered [ 4789.792551] LNet: 321316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4789.797361] LNetError: 321316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4867.622343] LNet: Removed LNI 192.168.201.117@tcp [ 4867.891977] Key type .llcrypt unregistered [ 4867.892870] Key type ._llcrypt unregistered [ 4875.130565] Key type ._llcrypt registered [ 4875.131528] Key type .llcrypt registered [ 4875.171342] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4882.813465] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4883.116159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4883.165476] alg: No test for adler32 (adler32-zlib) [ 4884.013654] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4884.108352] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4885.680189] Key type lgssc registered [ 4886.241424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4899.589889] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:36:20 (1784417780) [ 4902.594600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4902.605285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.711921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4903.725507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4903.780267] Lustre: lustre-MDT0000: new disk, initializing [ 4903.838895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4903.848952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4905.584390] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4911.006648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.077603] Lustre: 324515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4911.117421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4911.120971] Lustre: Skipped 1 previous similar message [ 4911.174864] Lustre: lustre-MDT0001: new disk, initializing [ 4911.220201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4911.242064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4911.248197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4912.902714] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4917.609815] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4920.243454] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4923.332507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4923.464746] Lustre: lustre-OST0000: new disk, initializing [ 4923.467710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4923.472903] Lustre: 326483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4923.521844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4926.105392] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4930.562582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4930.568090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4930.611829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4930.979167] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4940.773510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.775153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.776857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.778505] LustreError: Skipped 1 previous similar message [ 4940.785903] Lustre: Skipped 1 previous similar message [ 4944.235434] Lustre: server umount lustre-OST0000 complete [ 4946.913526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4946.918214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.926400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.932353] Lustre: Skipped 1 previous similar message [ 4951.010394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.011283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.245727] Lustre: server umount lustre-MDT0000 complete [ 4953.898890] LustreError: 324507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417835 with bad export cookie 4843880628446634466 [ 4953.905439] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.905855] LustreError: 324507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4954.048407] Lustre: server umount lustre-MDT0001 complete [ 4958.973603] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4960.454568] Key type lgssc unregistered [ 4960.618600] LNet: 328388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4960.623648] LNetError: 328388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4960.636575] LNet: Removed LNI 192.168.201.117@tcp [ 4961.032605] Key type .llcrypt unregistered [ 4961.034788] Key type ._llcrypt unregistered [ 4971.584596] Key type ._llcrypt registered [ 4971.586425] Key type .llcrypt registered [ 4971.649447] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4971.976051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4971.986216] alg: No test for adler32 (adler32-zlib) [ 4972.839728] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4972.924565] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4974.504144] Key type lgssc registered [ 4975.063295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4978.742637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4978.753654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.924123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.605662] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4984.980203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.187120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4986.556678] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4989.747873] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4991.448426] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4993.881805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.063130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.544312] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4998.982596] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5013.477551] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5014.090776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5016.033364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5016.037104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.044050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5019.861842] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 5020.646230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.966785] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 5024.969094] Lustre: Skipped 3 previous similar messages [ 5030.085748] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 5030.088743] Lustre: Skipped 3 previous similar messages [ 5030.368119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5030.437532] Lustre: server umount lustre-MDT0000 complete [ 5030.882165] LustreError: 329639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5030.887592] LustreError: 329639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5031.721459] LustreError: 329620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417913 with bad export cookie 14407583507333293633 [ 5031.726308] LustreError: 329620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5031.726543] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.205894] LustreError: 329635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.212736] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [ 5036.001891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.007052] Lustre: Skipped 1 previous similar message [ 5037.024985] LustreError: 329636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.031760] LustreError: 329636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5040.326320] LustreError: 329635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.446175] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [ 5045.446176] LustreError: 331479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.446189] LustreError: 331479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5045.451371] Lustre: Skipped 4 previous similar messages [ 5046.240122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5046.345946] Lustre: server umount lustre-MDT0001 complete [ 5048.054173] Lustre: server umount lustre-OST0000 complete [ 5051.694178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.065313] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5054.218562] Lustre: 333838:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.117@tcp [ 5055.204943] Lustre: server umount MGS complete [ 5059.124680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.322717] LustreError: 334409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.355713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.932910] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5064.945154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.110626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5066.717179] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5070.431301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.072882] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5076.222612] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5076.784751] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 5078.990856] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5087.850201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 5089.253225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5089.253596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.255486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.269133] Lustre: Skipped 1 previous similar message [ 5095.205192] Lustre: server umount lustre-OST0000 complete [ 5099.491044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.264161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5111.348311] Lustre: server umount lustre-MDT0000 complete [ 5112.950688] LustreError: 336804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417994 with bad export cookie 14407583507333294473 [ 5112.955996] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5112.958051] LustreError: 336804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5113.104506] Lustre: server umount lustre-MDT0001 complete [ 5117.839061] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5119.177692] Key type lgssc unregistered [ 5119.347642] LNet: 337843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5119.352299] LNetError: 337843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5119.361622] LNet: Removed LNI 192.168.201.117@tcp [ 5119.823731] Key type .llcrypt unregistered [ 5119.825635] Key type ._llcrypt unregistered [ 5127.657246] Key type ._llcrypt registered [ 5127.658673] Key type .llcrypt registered [ 5127.696302] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5135.123415] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5135.430274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5135.461056] alg: No test for adler32 (adler32-zlib) [ 5136.325667] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5136.423727] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5138.000235] Key type lgssc registered [ 5138.497851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5154.634077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5154.643843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.737627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5155.747385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5155.792368] Lustre: lustre-MDT0000: new disk, initializing [ 5155.813956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5155.820591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5157.484423] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5162.886243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.935970] Lustre: 340978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5162.977413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5162.981528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5162.985252] Lustre: Skipped 1 previous similar message [ 5163.026586] Lustre: lustre-MDT0001: new disk, initializing [ 5163.064158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5163.079122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5163.085491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5164.620515] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5169.328230] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.804486] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5174.215557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5174.353713] Lustre: lustre-OST0000: new disk, initializing [ 5174.355799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5174.359847] Lustre: 342947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5174.401454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.343957] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5180.416725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5180.421578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5180.472162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5180.647782] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5182.268642] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5182.381555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5183.764930] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5183.821967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5185.510098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5185.510568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.523684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.527165] Lustre: Skipped 1 previous similar message [ 5186.667780] Lustre: server umount lustre-OST0000 complete [ 5188.577287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5188.580994] LustreError: Skipped 1 previous similar message [ 5188.583248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.588809] Lustre: Skipped 1 previous similar message [ 5188.591497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.626490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.627492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.611446] Lustre: server umount lustre-MDT0000 complete [ 5196.259214] LustreError: 340969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418077 with bad export cookie 2824714282999460086 [ 5196.265461] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.265835] LustreError: 340969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5196.432936] Lustre: server umount lustre-MDT0001 complete [ 5200.183850] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:41:21 (1784418081) [ 5207.435697] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5210.636980] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5222.945967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.019799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5223.028353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5223.060996] Lustre: lustre-MDT0000: new disk, initializing [ 5223.079362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.084470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5224.253453] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5228.041755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.079404] Lustre: 347699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5228.099141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5228.100640] Lustre: Skipped 1 previous similar message [ 5228.133661] Lustre: lustre-MDT0001: new disk, initializing [ 5228.168196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.182307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5228.185209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5229.367591] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5232.789081] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.367332] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5236.382840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.478954] Lustre: lustre-OST0000: new disk, initializing [ 5236.480385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5236.482605] Lustre: 349665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5236.511323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.189455] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5242.122604] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5244.917697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5244.920698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5244.945931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5260.258300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5260.258557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.260311] LustreError: Skipped 1 previous similar message [ 5260.260847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5260.260851] Lustre: Skipped 1 previous similar message [ 5260.263013] Lustre: Skipped 1 previous similar message [ 5262.027364] Lustre: server umount lustre-OST0000 complete [ 5263.111853] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5265.377869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.380824] Lustre: Skipped 1 previous similar message [ 5269.276696] Lustre: server umount lustre-MDT0000 complete [ 5270.497900] LustreError: 347709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.507072] LustreError: 347709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5271.023158] LustreError: 348493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418152 with bad export cookie 2824714282999460975 [ 5271.030034] LustreError: 348493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5271.030263] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5271.169207] Lustre: server umount lustre-MDT0001 complete [ 5275.775456] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5277.229961] Key type lgssc unregistered [ 5277.368697] LNet: 351811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5277.373586] LNetError: 351811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5277.382571] LNet: Removed LNI 192.168.201.117@tcp [ 5277.853145] Key type .llcrypt unregistered [ 5277.854951] Key type ._llcrypt unregistered [ 5317.589058] Key type ._llcrypt registered [ 5317.590713] Key type .llcrypt registered [ 5317.644590] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5317.813582] Key type .llcrypt unregistered [ 5317.815827] Key type ._llcrypt unregistered [ 5352.898806] Key type ._llcrypt registered [ 5352.900299] Key type .llcrypt registered [ 5352.948513] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5353.095266] Key type .llcrypt unregistered [ 5353.096487] Key type ._llcrypt unregistered [ 5364.902752] Key type ._llcrypt registered [ 5364.903960] Key type .llcrypt registered [ 5364.936622] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5365.022175] Key type .llcrypt unregistered [ 5365.023190] Key type ._llcrypt unregistered [ 5377.365164] Key type ._llcrypt registered [ 5377.366162] Key type .llcrypt registered [ 5377.407726] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5377.566583] Key type .llcrypt unregistered [ 5377.568555] Key type ._llcrypt unregistered [ 5409.723470] Key type ._llcrypt registered [ 5409.724670] Key type .llcrypt registered [ 5409.760715] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5409.873545] Key type .llcrypt unregistered [ 5409.875251] Key type ._llcrypt unregistered [ 5422.835869] Key type ._llcrypt registered [ 5422.837939] Key type .llcrypt registered [ 5422.898240] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5423.061312] Key type .llcrypt unregistered [ 5423.063471] Key type ._llcrypt unregistered [ 5436.887402] Key type ._llcrypt registered [ 5436.888560] Key type .llcrypt registered [ 5436.930310] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5437.082319] Key type .llcrypt unregistered [ 5437.084167] Key type ._llcrypt unregistered [ 5450.955215] Key type ._llcrypt registered [ 5450.956467] Key type .llcrypt registered [ 5450.995233] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5451.103299] Key type .llcrypt unregistered [ 5451.104781] Key type ._llcrypt unregistered [ 5464.822780] Key type ._llcrypt registered [ 5464.824151] Key type .llcrypt registered [ 5464.873181] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5465.013501] Key type .llcrypt unregistered [ 5465.015209] Key type ._llcrypt unregistered [ 5479.186039] Key type ._llcrypt registered [ 5479.187367] Key type .llcrypt registered [ 5479.233883] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5479.379444] Key type .llcrypt unregistered [ 5479.381191] Key type ._llcrypt unregistered [ 5507.058114] Key type ._llcrypt registered [ 5507.062249] Key type .llcrypt registered [ 5507.121371] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5507.293378] Key type .llcrypt unregistered [ 5507.295485] Key type ._llcrypt unregistered [ 5520.479486] Key type ._llcrypt registered [ 5520.480787] Key type .llcrypt registered [ 5520.532372] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5520.687311] Key type .llcrypt unregistered [ 5520.689412] Key type ._llcrypt unregistered [ 5535.107190] Key type ._llcrypt registered [ 5535.108181] Key type .llcrypt registered [ 5535.146458] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5535.255596] Key type .llcrypt unregistered [ 5535.256460] Key type ._llcrypt unregistered [ 5560.501403] Key type ._llcrypt registered [ 5560.502432] Key type .llcrypt registered [ 5560.534582] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5568.951339] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5569.271211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5569.318329] alg: No test for adler32 (adler32-zlib) [ 5570.178979] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5570.260765] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5571.832178] Key type lgssc registered [ 5572.382420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5588.560686] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5588.565264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.648271] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5589.657924] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5589.694309] Lustre: tst250fs-MDT0000: new disk, initializing [ 5589.714711] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.720406] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5591.335298] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5596.380278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5596.454238] Lustre: 371736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5596.496836] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5596.500797] Lustre: Skipped 1 previous similar message [ 5596.549304] Lustre: tst250fs-MDT0001: new disk, initializing [ 5596.584828] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.601421] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5596.606819] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5598.202995] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5600.781548] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5603.862450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.984331] Lustre: tst250fs-OST0000: new disk, initializing [ 5603.986777] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5603.990215] Lustre: 373366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5604.026879] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.270562] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5609.482719] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5609.488962] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5609.552805] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5611.338304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5611.450083] Lustre: tst250fs-OST0001: new disk, initializing [ 5611.454108] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5611.459434] Lustre: 374239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5611.513751] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5614.221488] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5617.433516] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5618.227348] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5618.232486] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5618.278966] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5632.481501] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5632.486234] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5632.494491] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.508536] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.508780] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.518143] Lustre: Skipped 1 previous similar message [ 5633.522485] Lustre: Skipped 1 previous similar message [ 5635.489003] Lustre: server umount tst250fs-MDT0000 complete [ 5636.971134] LustreError: 371726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418518 with bad export cookie 17779869408382764020 [ 5636.978325] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.980467] LustreError: 371726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5637.099988] Lustre: server umount tst250fs-MDT0001 complete [ 5649.107834] Lustre: server umount tst250fs-OST0000 complete [ 5661.051272] Lustre: server umount tst250fs-OST0001 complete [ 5669.431460] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5672.987148] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5687.354325] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:49:28 (1784418568) [ 5687.938556] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5695.251417] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5699.393975] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5713.824725] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5571 sec ========== 19:49:54 (1784418594) [ 5714.728646] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:49:55 (1784418595) === [ 5715.955684] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:49:57 (1784418597) === [ 5732.517775] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5733.983411] Key type lgssc unregistered [ 5734.154609] LNet: 383381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5734.159201] LNetError: 383381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5734.167565] LNet: Removed LNI 192.168.201.117@tcp [ 5734.626166] Key type .llcrypt unregistered [ 5734.628340] Key type ._llcrypt unregistered