[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1128417417 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003273] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008598] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011015] pid_max: default: 32768 minimum: 301 [ 0.012134] LSM: Security Framework initializing [ 0.014011] Yama: becoming mindful. [ 0.015041] SELinux: Initializing. [ 0.016081] *** VALIDATE selinux *** [ 0.026637] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031518] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034118] *** VALIDATE tmpfs *** [ 0.036461] *** VALIDATE proc *** [ 0.037237] *** VALIDATE cgroup *** [ 0.038010] *** VALIDATE cgroup2 *** [ 0.039258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043027] Spectre V2 : User space: Vulnerable [ 0.044008] Speculative Store Bypass: Vulnerable [ 0.047248] debug: unmapping init [mem 0xffffffffab659000-0xffffffffab660fff] [ 0.050000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051021] ... version: 2 [ 0.052011] ... bit width: 48 [ 0.053014] ... generic registers: 4 [ 0.054021] ... value mask: 0000ffffffffffff [ 0.055013] ... max period: 00007fffffffffff [ 0.056013] ... fixed-purpose events: 3 [ 0.057018] ... event mask: 000000070000000f [ 0.058288] rcu: Hierarchical SRCU implementation. [ 0.061046] smp: Bringing up secondary CPUs ... [ 0.064064] x86: Booting SMP configuration: [ 0.065021] .... node #0, CPUs: #1 #2 #3 [ 0.081160] smp: Brought up 1 node, 4 CPUs [ 0.083021] smpboot: Max logical packages: 1 [ 0.084030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.167607] node 0 deferred pages initialised in 79ms [ 0.172180] devtmpfs: initialized [ 0.173354] x86/mm: Memory block size: 128MB [ 0.176762] gcov: version magic: 0x41383552 [ 0.178752] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.179209] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.180634] pinctrl core: initialized pinctrl subsystem [ 0.181387] [ 0.182012] ************************************************************* [ 0.183019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184018] ** ** [ 0.185024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.186019] ** ** [ 0.187023] ** This means that this kernel is built to expose internal ** [ 0.188021] ** IOMMU data structures, which may compromise security on ** [ 0.189021] ** your system. ** [ 0.190018] ** ** [ 0.192000] ** If you see this message and you are not debugging the ** [ 0.192018] ** kernel, report this immediately to your vendor! ** [ 0.193017] ** ** [ 0.194017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195020] ************************************************************* [ 0.197304] NET: Registered protocol family 16 [ 0.198552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202810] cpuidle: using governor menu [ 0.206540] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211261] PCI: Using configuration type 1 for base access [ 0.214247] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226581] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234173] cryptd: max_cpu_qlen set to 1000 [ 0.243671] ACPI: Added _OSI(Module Device) [ 0.244033] ACPI: Added _OSI(Processor Device) [ 0.248024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.259066] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.276060] ACPI: Interpreter enabled [ 0.277135] ACPI: PM: (supports S0 S3 S4 S5) [ 0.278023] ACPI: Using IOAPIC for interrupt routing [ 0.279206] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.280770] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302103] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.304025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.305103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308321] acpiphp: Slot [2] registered [ 0.309221] acpiphp: Slot [5] registered [ 0.310228] acpiphp: Slot [6] registered [ 0.311209] acpiphp: Slot [7] registered [ 0.312152] acpiphp: Slot [8] registered [ 0.313445] acpiphp: Slot [9] registered [ 0.315006] acpiphp: Slot [10] registered [ 0.316156] acpiphp: Slot [3] registered [ 0.317204] acpiphp: Slot [4] registered [ 0.318149] acpiphp: Slot [11] registered [ 0.319256] acpiphp: Slot [12] registered [ 0.320155] acpiphp: Slot [13] registered [ 0.321119] acpiphp: Slot [14] registered [ 0.322134] acpiphp: Slot [15] registered [ 0.323136] acpiphp: Slot [16] registered [ 0.324131] acpiphp: Slot [17] registered [ 0.325168] acpiphp: Slot [18] registered [ 0.326112] acpiphp: Slot [19] registered [ 0.327127] acpiphp: Slot [20] registered [ 0.328137] acpiphp: Slot [21] registered [ 0.329395] acpiphp: Slot [22] registered [ 0.330116] acpiphp: Slot [23] registered [ 0.331204] acpiphp: Slot [24] registered [ 0.333108] acpiphp: Slot [25] registered [ 0.334170] acpiphp: Slot [26] registered [ 0.335168] acpiphp: Slot [27] registered [ 0.336166] acpiphp: Slot [28] registered [ 0.337187] acpiphp: Slot [29] registered [ 0.338763] acpiphp: Slot [30] registered [ 0.339199] acpiphp: Slot [31] registered [ 0.340110] PCI host bridge to bus 0000:00 [ 0.341035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347369] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350429] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353650] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366460] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371107] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374161] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378449] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380755] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.387028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398572] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.404031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.417433] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.420027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423035] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.430026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.435992] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.439023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.442028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.449030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.456018] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.459038] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.462034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.469032] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.475547] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.478094] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.481031] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.489029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.495461] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.498026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.501036] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.508031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.516191] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.517604] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.518558] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.520189] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.522069] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.529211] iommu: Default domain type: Passthrough [ 0.535159] SCSI subsystem initialized [ 0.541168] ACPI: bus type USB registered [ 0.545154] usbcore: registered new interface driver usbfs [ 0.548158] usbcore: registered new interface driver hub [ 0.553089] usbcore: registered new device driver usb [ 0.556213] pps_core: LinuxPPS API ver. 1 registered [ 0.558018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.565114] PTP clock support registered [ 0.566205] EDAC MC: Ver: 3.0.0 [ 0.572326] PCI: Using ACPI for IRQ routing [ 0.578786] NetLabel: Initializing [ 0.580023] NetLabel: domain hash size = 128 [ 0.584018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.589125] NetLabel: unlabeled traffic allowed by default [ 0.596174] vgaarb: loaded [ 0.599752] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.603018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.617153] clocksource: Switched to clocksource kvm-clock [ 1.079578] VFS: Disk quotas dquot_6.6.0 [ 1.087409] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.101451] *** VALIDATE ramfs *** [ 1.110542] *** VALIDATE hugetlbfs *** [ 1.113926] pnp: PnP ACPI init [ 1.121261] pnp: PnP ACPI: found 6 devices [ 1.158184] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.162165] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.164672] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.166947] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.168805] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.171083] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.173357] NET: Registered protocol family 2 [ 1.175835] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.179832] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.183272] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.188432] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.192245] TCP: Hash tables configured (established 65536 bind 65536) [ 1.194839] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.197811] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.200521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.203452] NET: Registered protocol family 1 [ 1.206380] RPC: Registered named UNIX socket transport module. [ 1.208240] RPC: Registered udp transport module. [ 1.210174] RPC: Registered tcp transport module. [ 1.211829] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.214308] NET: Registered protocol family 44 [ 1.215992] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.217871] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.222106] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.226548] PCI: CLS 0 bytes, default 64 [ 1.228192] Unpacking initramfs... [ 4.263257] debug: unmapping init [mem 0xffff8c133cc54000-0xffff8c133ffbffff] [ 4.271176] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.273944] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 4.278703] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.068216] Initialise system trusted keyrings [ 5.070055] Key type blacklist registered [ 5.073760] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.100269] zbud: loaded [ 5.113891] *** VALIDATE nfs *** [ 5.122339] *** VALIDATE nfs4 *** [ 5.124357] pstore: using deflate compression [ 5.128880] Platform Keyring initialized [ 5.328661] NET: Registered protocol family 38 [ 5.332572] Key type asymmetric registered [ 5.336160] Asymmetric key parser 'x509' registered [ 5.340261] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.347252] io scheduler mq-deadline registered [ 5.350084] io scheduler kyber registered [ 5.353896] io scheduler bfq registered [ 5.357434] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.366968] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.376076] hrtimer: interrupt took 3555530 ns [ 5.400389] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.408898] ACPI: Power Button [PWRF] [ 5.423740] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.443268] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.487638] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.499932] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.563141] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.591491] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.622077] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.626897] Non-volatile memory driver v1.3 [ 5.628436] Linux agpgart interface v0.103 [ 5.661977] virtio_blk virtio1: [vda] 134632 512-byte logical blocks (68.9 MB/65.7 MiB) [ 5.665162] vda: detected capacity change from 0 to 68931584 [ 5.680804] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.684275] vdb: detected capacity change from 0 to 1073741824 [ 5.707581] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.714828] vdc: detected capacity change from 0 to 2621440000 [ 5.751016] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.754458] vdd: detected capacity change from 0 to 2621440000 [ 5.770659] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.774194] vde: detected capacity change from 0 to 4294967296 [ 5.789466] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.796341] vdf: detected capacity change from 0 to 4294967296 [ 5.809784] libphy: Fixed MDIO Bus: probed [ 5.821124] usbcore: registered new interface driver usbserial_generic [ 5.827267] usbserial: USB Serial support registered for generic [ 5.832972] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.842102] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.846558] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.851765] mousedev: PS/2 mouse device common for all mice [ 5.858427] rtc_cmos 00:05: RTC can wake from S4 [ 5.866562] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.871366] rtc_cmos 00:05: registered as rtc0 [ 5.882708] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.886385] intel_pstate: CPU model not supported [ 5.897332] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.897869] hid: raw HID events driver (C) Jiri Kosina [ 5.917496] usbcore: registered new interface driver usbhid [ 5.925936] usbhid: USB HID core driver [ 5.928305] drop_monitor: Initializing network drop monitor service [ 5.935208] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.945549] Initializing XFRM netlink socket [ 5.954610] NET: Registered protocol family 10 [ 5.961164] Segment Routing with IPv6 [ 5.966802] NET: Registered protocol family 17 [ 5.970831] mpls_gso: MPLS GSO support [ 5.982590] RAS: Correctable Errors collector initialized. [ 5.995496] AVX version of gcm_enc/dec engaged. [ 6.000438] AES CTR mode by8 optimization enabled [ 6.292251] sched_clock: Marking stable (6292208171, 0)->(9228695281, -2936487110) [ 6.299524] registered taskstats version 1 [ 6.307815] Loading compiled-in X.509 certificates [ 6.313310] zswap: loaded using pool lzo/zbud [ 6.405488] Key type big_key registered [ 6.442138] Key type encrypted registered [ 6.444339] ima: No TPM chip found, activating TPM-bypass! [ 6.446575] ima: Allocated hash algorithm: sha1 [ 6.448550] ima: No architecture policies found [ 6.450551] evm: Initialising EVM extended attributes: [ 6.452772] evm: security.selinux [ 6.453921] evm: security.ima [ 6.454949] evm: security.capability [ 6.456381] evm: HMAC attrs: 0x1 [ 6.459606] rtc_cmos 00:05: setting system clock to 2026-05-04 15:52:06 UTC (1777909926) [ 6.468409] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 6.472293] debug: unmapping init [mem 0xffffffffab382000-0xffffffffab658fff] [ 6.483186] Write protecting the kernel read-only data: 28672k [ 6.486925] debug: unmapping init [mem 0xffffffffa9a03000-0xffffffffa9bfffff] [ 6.489467] debug: unmapping init [mem 0xffffffffaa314000-0xffffffffaa3fffff] [ 6.599242] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.617525] systemd[1]: Detected virtualization kvm. [ 6.622482] systemd[1]: Detected architecture x86-64. [ 6.624585] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.661169] systemd[1]: No hostname configured. [ 6.665533] systemd[1]: Set hostname to . [ 6.668948] random: systemd: uninitialized urandom read (16 bytes read) [ 6.674195] systemd[1]: Initializing machine ID from random generator. [ 6.932298] random: ln: uninitialized urandom read (6 bytes read) [ 7.950749] random: systemd: uninitialized urandom read (16 bytes read) [ 7.970423] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.998467] random: systemd: uninitialized urandom read (16 bytes read) [ 8.007857] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 8.029358] random: systemd: uninitialized urandom read (16 bytes read) [ 8.071674] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.637890] device-mapper: uevent: version 1.0.3 [ 12.642337] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 15.237621] random: fast init done [ 15.301327] scsi host0: ata_piix [ 15.322214] scsi host1: ata_piix [ 15.324196] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.326757] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.073604] virtio_net virtio0 ens2: renamed from eth0 [ 22.116742] random: crng init done [ 22.118848] random: 5 urandom warning(s) missed due to ratelimiting [ 25.445841] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 27.652055] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.227557] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.435451] SELinux: Disabled at runtime. [ 33.583758] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 33.628409] systemd[1]: Detected virtualization kvm. [ 33.636309] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 35.102495] systemd[1]: initrd-switch-root.service: Succeeded. [ 35.111194] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 35.120183] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 35.125885] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 35.137452] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 35.154428] systemd[1]: Starting Journal Service... Starting Journal Service... [ 35.166157] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ 35.493492] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 36.732710] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.705866] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.951619] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.392037] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.466726] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 44.774266] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 45.629733] NFS: Registering the id_resolver key type [ 45.631203] Key type id_resolver registered [ 45.634874] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg222-server login: [ 78.192708] libcfs: loading out-of-tree module taints kernel. [ 78.220527] Key type ._llcrypt registered [ 78.222927] Key type .llcrypt registered [ 78.273262] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 87.298810] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 87.987515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.996592] alg: No test for adler32 (adler32-zlib) [ 89.105878] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 89.541706] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 91.207179] Key type lgssc registered [ 92.128060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.300168] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.171365] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 128.638899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 128.650752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.776228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.792859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.834904] Lustre: lustre-MDT0000: new disk, initializing [ 129.883230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.897802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 131.756528] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.558060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.607516] Lustre: 6482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 138.630667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 138.634617] Lustre: Skipped 1 previous similar message [ 138.686351] Lustre: lustre-MDT0001: new disk, initializing [ 138.722479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 138.738838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 138.742112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 140.467690] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.806536] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 149.627922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 149.771789] Lustre: lustre-OST0000: new disk, initializing [ 149.774725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 149.810352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.249026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 151.254094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 151.305282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 153.131304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.420093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 161.487317] Lustre: lustre-OST0001: new disk, initializing [ 161.491114] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 161.525531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.460480] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.929789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 167.934506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 172.578357] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 173.045321] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 177.506413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 181.362411] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing check_logdir /tmp/testlogs/ [ 184.511152] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing yml_node [ 186.739944] Lustre: DEBUG MARKER: Client: 2.17.52.127 [ 188.409398] Lustre: DEBUG MARKER: MDS: 2.17.52.127 [ 189.957996] Lustre: DEBUG MARKER: OSS: 2.17.52.127 [ 191.117836] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 4 11:55:10 EDT 2026 [ 201.861703] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 202.789638] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 215.529765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 215.536687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.545855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.105124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.106813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.122121] Lustre: Skipped 1 previous similar message [ 219.132109] Lustre: Skipped 3 previous similar messages [ 223.802520] Lustre: server umount lustre-MDT0000 complete [ 225.731943] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910145 with bad export cookie 7321458495102033673 [ 225.734620] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 225.736046] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 229.346201] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.347695] LustreError: 6488:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 229.355249] Lustre: Skipped 1 previous similar message [ 229.355806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 229.363226] LustreError: 6488:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 229.366426] Lustre: Skipped 1 previous similar message [ 230.369259] LustreError: 7417:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 230.378953] LustreError: 7417:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 235.494024] LustreError: 9992:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.495951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 235.514906] LustreError: 9992:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 235.536250] Lustre: Skipped 3 previous similar messages [ 240.095161] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 240.222951] Lustre: server umount lustre-MDT0001 complete [ 242.473307] Lustre: server umount lustre-OST0000 complete [ 244.580705] Lustre: server umount lustre-OST0001 complete [ 248.337608] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 253.457959] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 274.172795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.329198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 274.348238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 274.417166] Lustre: lustre-MDT0000: new disk, initializing [ 274.464457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.478494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 276.772558] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 282.781081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.840812] Lustre: 14947:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 282.873427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 282.876610] Lustre: Skipped 1 previous similar message [ 282.934971] Lustre: lustre-MDT0001: new disk, initializing [ 282.995907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 283.017210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 283.023955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 285.201081] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 289.623146] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 291.482489] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 294.746530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.033995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 296.082619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 297.618976] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 301.541924] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 303.434817] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 303.529282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 305.539433] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 305.638394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 306.659875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 306.660983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.662413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 306.666917] LustreError: Skipped 1 previous similar message [ 306.676635] Lustre: Skipped 2 previous similar messages [ 312.847612] Lustre: server umount lustre-OST0000 complete [ 316.902170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.904082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.909686] Lustre: Skipped 1 previous similar message [ 316.917823] Lustre: Skipped 4 previous similar messages [ 320.691626] Lustre: server umount lustre-MDT0000 complete [ 322.017737] LustreError: 14958:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.023086] LustreError: 14958:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 322.457199] LustreError: 15724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910242 with bad export cookie 7321458495102035213 [ 322.457199] LustreError: 14940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910242 with bad export cookie 7321458495102035213 [ 322.459073] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.462776] LustreError: 14940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 322.615910] Lustre: server umount lustre-MDT0001 complete [ 325.442261] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:57:24 (1777910244) [ 335.057091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 335.264532] LustreError: 19412:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.295295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 335.299959] Lustre: Skipped 1 previous similar message [ 337.206765] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 341.378314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.341125] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 346.115321] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 347.860224] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 348.915561] Lustre: Failing over lustre-MDT0000 [ 349.039268] Lustre: server umount lustre-MDT0000 complete [ 351.008442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.009224] LustreError: 19407:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.015958] Lustre: Skipped 1 previous similar message [ 351.025338] LustreError: 19407:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 363.796571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.855896] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.052889] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 366.058034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 366.070297] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 366.216265] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 370.897084] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 376.288245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 376.293727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.301138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.729234] Lustre: server umount lustre-MDT0000 complete [ 382.080142] LustreError: 19392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910302 with bad export cookie 7321458495102036627 [ 382.084729] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.095116] LustreError: 19392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 382.323438] Lustre: server umount lustre-MDT0001 complete [ 387.736706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.000219] LustreError: 22407:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.008262] LustreError: 22407:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 388.061467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.065328] Lustre: Skipped 2 previous similar messages [ 390.625903] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 396.220565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.857432] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 402.026374] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 404.429903] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 405.615865] Lustre: Failing over lustre-MDT0000 [ 405.725363] Lustre: server umount lustre-MDT0000 complete [ 420.417516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.489058] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.665313] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 421.668228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 421.671035] Lustre: Skipped 1 previous similar message [ 421.675867] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 422.253378] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 426.370318] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 432.097861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.098601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 432.099232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 432.099237] Lustre: Skipped 4 previous similar messages [ 432.103170] Lustre: Skipped 4 previous similar messages [ 443.359137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 443.427707] Lustre: server umount lustre-MDT0000 complete [ 445.138736] LustreError: 22387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910365 with bad export cookie 7321458495102037565 [ 445.139163] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.143247] LustreError: 22387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 445.293328] Lustre: server umount lustre-MDT0001 complete [ 449.020915] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:59:28 (1777910368) [ 456.418709] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 460.493774] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 478.423947] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 483.149421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.262216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 483.276719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 483.279303] Lustre: Skipped 1 previous similar message [ 483.317229] Lustre: lustre-MDT0000: new disk, initializing [ 483.319239] Lustre: Skipped 1 previous similar message [ 483.362543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.365159] Lustre: Skipped 2 previous similar messages [ 483.371699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 483.374917] Lustre: Skipped 1 previous similar message [ 485.224544] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 490.053500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.099797] Lustre: 28378:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 490.210073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 491.880151] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 496.074801] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 498.039853] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 501.298837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.869517] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 502.893363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 502.899584] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 503.832410] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 507.750307] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 511.869442] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 523.233519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.240172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.246951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.249790] Lustre: Skipped 5 previous similar messages [ 525.490648] Lustre: server umount lustre-OST0000 complete [ 533.607645] Lustre: server umount lustre-MDT0000 complete [ 533.989885] LustreError: 29389:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.997230] LustreError: 29389:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 535.292147] LustreError: 28368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910455 with bad export cookie 7321458495102037964 [ 535.296145] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.300454] LustreError: 28368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.475590] Lustre: server umount lustre-MDT0001 complete [ 539.842958] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 541.128660] Key type lgssc unregistered [ 541.280563] LNet: 31941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.286256] LNetError: 31941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.296476] LNet: Removed LNI 192.168.202.122@tcp [ 541.704130] Key type .llcrypt unregistered [ 541.705558] Key type ._llcrypt unregistered [ 552.069315] Key type ._llcrypt registered [ 552.070912] Key type .llcrypt registered [ 552.122525] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 552.502802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.553344] alg: No test for adler32 (adler32-zlib) [ 553.422485] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 553.512374] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 555.095166] Key type lgssc registered [ 555.506997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.818179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 558.825900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.044703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.585617] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 565.258202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.431450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 567.008456] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 569.275339] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 570.895752] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 574.131995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.334113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.405283] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 577.081804] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 579.493859] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 596.027259] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:01:55 (1777910515) [ 596.988385] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 597.904538] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 601.171228] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:02:00 (1777910520) [ 605.073670] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:02:04 (1777910524) [ 605.582633] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 609.557671] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 609.561170] Lustre: Skipped 2 previous similar messages [ 610.278695] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:02:09 (1777910529) [ 613.885989] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 613.892665] Lustre: Skipped 4 previous similar messages [ 621.309360] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:02:20 (1777910540) [ 622.671907] Lustre: *** cfs_fail_loc=131b, val=2*** [ 624.414398] Lustre: *** cfs_fail_loc=131b, val=2*** [ 628.446640] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:02:27 (1777910547) [ 629.425796] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 629.428590] Lustre: Skipped 9 previous similar messages [ 636.319653] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:02:35 (1777910555) [ 647.137480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 647.137588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.141554] LustreError: Skipped 1 previous similar message [ 647.146776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 647.150243] Lustre: Skipped 1 previous similar message [ 649.459617] Lustre: server umount lustre-OST0000 complete [ 652.257357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.267365] Lustre: Skipped 1 previous similar message [ 652.271569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.275437] Lustre: Skipped 1 previous similar message [ 657.376571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.379414] Lustre: Skipped 1 previous similar message [ 657.588825] Lustre: server umount lustre-MDT0000 complete [ 659.381294] LustreError: 33307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910579 with bad export cookie 17982951716825084174 [ 659.384073] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.386505] LustreError: 33307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.562459] Lustre: server umount lustre-MDT0001 complete [ 664.296491] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 665.651560] Key type lgssc unregistered [ 665.804972] LNet: 40818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.808555] LNetError: 40818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.817456] LNet: Removed LNI 192.168.202.122@tcp [ 666.236320] Key type .llcrypt unregistered [ 666.238101] Key type ._llcrypt unregistered [ 674.225256] Key type ._llcrypt registered [ 674.226906] Key type .llcrypt registered [ 674.278821] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 681.931488] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 682.289966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.302514] alg: No test for adler32 (adler32-zlib) [ 683.184578] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 683.283983] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 684.887236] Key type lgssc registered [ 685.322088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.009889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 700.020826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.133086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 701.149225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 701.187825] Lustre: lustre-MDT0000: new disk, initializing [ 701.226151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.235758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 702.869979] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 708.253051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.304565] Lustre: 43910:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 708.328837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 708.331189] Lustre: Skipped 1 previous similar message [ 708.369435] Lustre: lustre-MDT0001: new disk, initializing [ 708.412302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 708.430966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 708.436444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 710.107794] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 714.000434] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.741915] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 718.791396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.907255] Lustre: lustre-OST0000: new disk, initializing [ 718.909503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 718.951745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.364868] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 725.088352] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 726.018459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 726.023492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 726.064345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 726.667175] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 726.752519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 728.280977] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 728.354835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 731.107743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 731.108627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.111720] LustreError: Skipped 1 previous similar message [ 731.118414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 731.121867] Lustre: Skipped 1 previous similar message [ 735.280603] Lustre: server umount lustre-OST0000 complete [ 739.297076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 739.301137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.307621] Lustre: Skipped 1 previous similar message [ 739.309904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.344343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.347260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.076524] Lustre: server umount lustre-MDT0000 complete [ 744.535072] LustreError: 43902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910664 with bad export cookie 7938908518190242762 [ 744.537943] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.538521] LustreError: 43902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 744.648683] Lustre: server umount lustre-MDT0001 complete [ 748.305252] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:04:27 (1777910667) [ 751.278050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.542126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.398649] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 756.705325] LustreError: 47959:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.711917] LustreError: 47959:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 757.634876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.809770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 759.538795] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 761.908763] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 763.646705] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 766.696361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 766.828640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.227622] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 771.460960] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 779.674946] LustreError: 50146:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 780.157689] Lustre: Modifying parameter general.timeout=1 in log params [ 781.453414] Lustre: Modifying parameter general.timeout=4 in log params [ 781.456478] Lustre: Skipped 2 previous similar messages [ 783.675742] Lustre: Modifying parameter general.timeout=9 in log params [ 783.678189] Lustre: Skipped 4 previous similar messages [ 787.967139] Lustre: Modifying parameter general.timeout=20 in log params [ 787.969296] Lustre: Skipped 10 previous similar messages [ 796.142498] Lustre: Modifying parameter general.timeout=44 in log params [ 796.144455] Lustre: Skipped 23 previous similar messages [ 812.509756] Lustre: Modifying parameter general.timeout=86 in log params [ 812.511562] Lustre: Skipped 41 previous similar messages [ 833.706317] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:05:53 (1777910753) [ 834.839421] Lustre: 57053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 836.254600] Lustre: 57250:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 836.261205] Lustre: 57250:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 838.337630] Lustre: 57546:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 838.342467] Lustre: 57546:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 845.018479] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 845.021963] Lustre: Skipped 62 previous similar messages [ 848.229696] Lustre: 58904:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 848.234346] LustreError: 58904:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 849.793660] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:06:09 (1777910769) [ 855.008352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 855.010837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.016559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.019152] Lustre: Skipped 1 previous similar message [ 860.249403] Lustre: server umount lustre-MDT0000 complete [ 861.576121] LustreError: 47939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910781 with bad export cookie 7938908518190243651 [ 861.579393] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.580810] LustreError: 47939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 861.692211] Lustre: server umount lustre-MDT0001 complete [ 873.503974] Lustre: server umount lustre-OST0000 complete [ 883.252582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.363069] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 883.489867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.957456] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 888.409215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.453499] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 888.466779] Lustre: 61428:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 890.035988] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 892.103418] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 893.544745] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.025873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.103045] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 898.367046] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 900.531196] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 906.242691] Lustre: 63657:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 911.841018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.842543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.844717] LustreError: Skipped 1 previous similar message [ 911.845203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.845206] Lustre: Skipped 3 previous similar messages [ 911.861972] Lustre: Skipped 3 previous similar messages [ 914.448115] Lustre: server umount lustre-OST0000 complete [ 922.080078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.082831] Lustre: Skipped 4 previous similar messages [ 930.271103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 930.336364] Lustre: server umount lustre-MDT0000 complete [ 931.764431] LustreError: 61420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910851 with bad export cookie 7938908518190246024 [ 931.766984] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.769387] LustreError: 61420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 931.885679] Lustre: server umount lustre-MDT0001 complete [ 935.873651] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 936.943659] Key type lgssc unregistered [ 937.077312] LNet: 64846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.080505] LNetError: 64846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.089359] LNet: Removed LNI 192.168.202.122@tcp [ 937.416304] Key type .llcrypt unregistered [ 937.417588] Key type ._llcrypt unregistered [ 947.950591] Key type ._llcrypt registered [ 947.951865] Key type .llcrypt registered [ 947.999457] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 948.400437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.412572] alg: No test for adler32 (adler32-zlib) [ 949.281700] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 949.371084] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 950.959547] Key type lgssc registered [ 951.351501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.269923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 954.279328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.465753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.995256] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 960.461791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.676430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 962.183616] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 964.242440] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 965.655996] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 968.146913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 968.347864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.489656] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 972.496153] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 987.217873] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 991.201867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.202993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.203760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.205836] Lustre: Skipped 1 previous similar message [ 996.184600] Lustre: server umount lustre-MDT0000 complete [ 996.319977] LustreError: 66220:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.329228] LustreError: 66220:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 997.630159] LustreError: 66200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910917 with bad export cookie 2738628475805928999 [ 997.633674] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.636195] LustreError: 66200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 997.763428] Lustre: server umount lustre-MDT0001 complete [ 1009.566353] Lustre: server umount lustre-OST0000 complete [ 1019.442274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.580110] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1019.593962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1019.714472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.240458] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1024.553561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.591057] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1024.600861] Lustre: 70765:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1024.767130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.249890] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1028.246083] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.684475] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1032.265642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.338193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1034.513676] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1036.429455] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.734859] Lustre: 73023:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1048.428821] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1052.644092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.644803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.647079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.647087] Lustre: Skipped 2 previous similar messages [ 1052.647338] LustreError: Skipped 1 previous similar message [ 1052.662488] Lustre: Skipped 2 previous similar messages [ 1055.887851] Lustre: server umount lustre-OST0000 complete [ 1057.760745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.769646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.772623] Lustre: Skipped 1 previous similar message [ 1062.879897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.881909] Lustre: Skipped 1 previous similar message [ 1063.523532] Lustre: server umount lustre-MDT0000 complete [ 1065.036344] LustreError: 70805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910985 with bad export cookie 2738628475805929811 [ 1065.038815] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1065.041906] LustreError: 70805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1065.192667] Lustre: server umount lustre-MDT0001 complete [ 1070.623610] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1072.022843] Key type lgssc unregistered [ 1072.178364] LNet: 74211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.181543] LNetError: 74211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.191355] LNet: Removed LNI 192.168.202.122@tcp [ 1072.545323] Key type .llcrypt unregistered [ 1072.547107] Key type ._llcrypt unregistered [ 1084.763861] Key type ._llcrypt registered [ 1084.765276] Key type .llcrypt registered [ 1084.817660] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1085.188109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.198470] alg: No test for adler32 (adler32-zlib) [ 1086.117878] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1086.221310] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1087.823137] Key type lgssc registered [ 1088.231857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1091.427439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.438500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.676875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.461122] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1098.418300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.679067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1100.493668] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1102.996587] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1104.703369] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1107.836288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.112996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.401361] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1112.833048] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1121.582582] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1122.282233] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1122.284114] Lustre: Skipped 1 previous similar message [ 1123.349451] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1123.352205] Lustre: Skipped 2 previous similar messages [ 1125.504577] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1125.507050] Lustre: Skipped 5 previous similar messages [ 1129.703392] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1129.707659] Lustre: Skipped 11 previous similar messages [ 1137.808578] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1137.810690] Lustre: Skipped 22 previous similar messages [ 1154.076159] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1154.078975] Lustre: Skipped 45 previous similar messages [ 1160.232974] Lustre: 83316:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1161.490511] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:11:20 (1777911080) [ 1162.182941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1162.984130] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:11:22 (1777911082) [ 1165.792184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.800593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.808549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.866410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.866738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.866866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.399277] Lustre: server umount lustre-OST0000 complete [ 1173.984827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.985877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.990523] Lustre: Skipped 1 previous similar message [ 1173.993266] Lustre: Skipped 1 previous similar message [ 1179.104070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.106941] Lustre: Skipped 1 previous similar message [ 1184.224359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.227207] Lustre: Skipped 1 previous similar message [ 1186.271124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1186.337485] Lustre: server umount lustre-MDT0000 complete [ 1187.842961] LustreError: 75617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911107 with bad export cookie 1370325565746494091 [ 1187.843654] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.846493] LustreError: 75617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1187.979711] Lustre: server umount lustre-MDT0001 complete [ 1192.134770] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1193.407205] Key type lgssc unregistered [ 1193.545381] LNet: 84547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.548270] LNetError: 84547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.560052] LNet: Removed LNI 192.168.202.122@tcp [ 1193.897327] Key type .llcrypt unregistered [ 1193.898862] Key type ._llcrypt unregistered [ 1195.390645] Key type ._llcrypt registered [ 1195.391834] Key type .llcrypt registered [ 1195.440823] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1197.132719] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules [ 1197.585345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.597036] alg: No test for adler32 (adler32-zlib) [ 1198.475962] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1198.572605] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1200.159170] Key type lgssc registered [ 1200.623074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1201.130144] LustreError: 85666:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 1208.794880] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1212.111634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1222.252584] LustreError: 85666:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1222.262384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.506713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.919265] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1227.966739] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:12:27 (1777911147) [ 1229.642926] LustreError: 86704:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1229.647677] LustreError: 86704:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1229.651687] LustreError: 86704:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1229.768506] Lustre: server umount lustre-MDT0000 complete [ 1234.631067] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1235.609763] Key type lgssc unregistered [ 1235.736518] LNet: 88160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.739402] LNetError: 88160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.748345] LNet: Removed LNI 192.168.202.122@tcp [ 1236.030097] Key type .llcrypt unregistered [ 1236.031462] Key type ._llcrypt unregistered [ 1243.497271] Key type ._llcrypt registered [ 1243.498656] Key type .llcrypt registered [ 1243.540614] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1243.866972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1243.880819] alg: No test for adler32 (adler32-zlib) [ 1244.741535] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1244.835786] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1246.431167] Key type lgssc registered [ 1246.834696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1249.741428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1249.748258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.904102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.190396] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1255.057356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.273475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.602390] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1258.373108] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1259.593537] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1262.003975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1262.198488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.121622] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1265.832321] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.782021] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:13:37 (1777911217) [ 1298.343897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1298.936306] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:13:38 (1777911218) [ 1301.473745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.474841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.478624] Lustre: Skipped 2 previous similar messages [ 1301.478920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.480351] Lustre: Skipped 2 previous similar messages [ 1306.070744] Lustre: server umount lustre-MDT0000 complete [ 1306.592354] LustreError: 91445:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.597430] LustreError: 91445:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1307.387546] LustreError: 89318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911227 with bad export cookie 1735866839623221207 [ 1307.391088] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.392103] LustreError: 89318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1307.520675] Lustre: server umount lustre-MDT0001 complete [ 1319.290971] Lustre: server umount lustre-OST0000 complete [ 1323.530442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.709473] LustreError: 93288:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.758967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.296704] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1328.705231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.998823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1330.565893] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1332.598722] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1334.004900] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1338.481570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1338.553268] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1338.556020] LustreError: 93277:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1338.564559] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1338.567556] LustreError: 95144:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1338.570471] LustreError: 95144:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1338.572685] LustreError: 95144:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1338.596599] Lustre: server umount lustre-OST0000 complete [ 1338.598190] LustreError: 95144:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1341.770071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1341.859787] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1341.863165] LustreError: 93278:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1341.873234] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1341.876483] LustreError: 95608:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1341.879358] LustreError: 95608:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1341.881656] LustreError: 95608:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1341.909544] Lustre: server umount lustre-OST0000 complete [ 1341.911543] LustreError: 95608:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1345.351577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.431693] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1345.434629] Lustre: Found index 0 for lustre-OST0000, updating log [ 1345.436955] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1345.469586] Lustre: lustre-OST0000: new disk, initializing [ 1345.472521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1345.566508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.613258] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1348.914961] Lustre: Failing over lustre-OST0000 [ 1348.978428] Lustre: server umount lustre-OST0000 complete [ 1353.441325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.446750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1353.452224] Lustre: Skipped 1 previous similar message [ 1353.455886] Lustre: Skipped 1 previous similar message [ 1358.816106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.818321] Lustre: Skipped 1 previous similar message [ 1360.889405] Lustre: server umount lustre-MDT0000 complete [ 1362.122897] LustreError: 93270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911282 with bad export cookie 1735866839623223937 [ 1362.124844] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.126543] LustreError: 93270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1362.231423] Lustre: server umount lustre-MDT0001 complete [ 1365.133692] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:14:44 (1777911284) [ 1370.514428] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1374.108779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.266508] LustreError: 98447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.301208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.649673] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1378.518864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.051736] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1381.054860] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1383.352135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.589588] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1384.610194] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1384.621297] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1384.621421] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1384.624389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1385.488995] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1388.434234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.465736] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1388.499033] Lustre: lustre-OST0001: new disk, initializing [ 1388.500895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1390.559487] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1391.807794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1391.811514] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1391.848779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1393.857514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1399.743275] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1401.966274] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 1402.338781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.343500] Lustre: Skipped 1 previous similar message [ 1407.078816] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 1407.081882] Lustre: Skipped 3 previous similar messages [ 1407.319875] Lustre: server umount lustre-MDT0001 complete [ 1407.456815] LustreError: 98447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.462471] LustreError: 98447:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1410.848282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.882086] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1410.884468] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1410.900749] Lustre: 98435:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1411.172074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.174648] Lustre: Skipped 3 previous similar messages [ 1412.583956] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1412.774586] LustreError: 102459:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1412.777335] LustreError: 102459:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1412.780274] LustreError: 98439:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1412.809664] LustreError: 102470:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.122@tcp: cfg command failed: rc = -17 [ 1412.813045] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1412.813045] [ 1415.848677] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 12:15:35 (1777911335) [ 1416.168196] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1421.795745] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1427.805869] Lustre: 104266:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1436.096536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1437.326470] Lustre: 105397:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1485.791735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.792417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.795495] Lustre: Skipped 3 previous similar messages [ 1485.797344] Lustre: Skipped 1 previous similar message [ 1489.568109] Lustre: server umount lustre-MDT0000 complete [ 1490.822306] LustreError: 100710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911410 with bad export cookie 1735866839623224882 [ 1490.824246] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.827076] LustreError: 100710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1490.911556] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.911612] LustreError: 101673:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.915101] Lustre: Skipped 2 previous similar messages [ 1490.921513] LustreError: 101673:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1490.945827] Lustre: server umount lustre-MDT0001 complete [ 1498.369573] Lustre: server umount lustre-OST0000 complete [ 1499.759739] Lustre: server umount lustre-OST0001 complete [ 1501.777513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1505.068211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1507.857389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1510.995932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1517.166177] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1520.623966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.629691] Lustre: lustre-MDT0000: reset Object Index mappings [ 1520.774323] LustreError: 109300:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.803926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.014845] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1524.778101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.786101] Lustre: lustre-MDT0001: reset Object Index mappings [ 1526.303636] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1527.240841] Lustre: 110414:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1529.333587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1530.597807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 1531.511465] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1534.341465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.593571] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1535.594684] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1535.594813] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1536.524353] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1539.735973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1540.990176] Lustre: 112252:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1556.098416] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 12:17:55 (1777911475) [ 1586.655849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1586.658812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.663239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.207044] Lustre: server umount lustre-MDT0000 complete [ 1592.289419] LustreError: 110020:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.293778] LustreError: 110020:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1592.397501] LustreError: 109283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911512 with bad export cookie 1735866839623271852 [ 1592.401261] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1592.403168] LustreError: 109283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1598.651606] Lustre: server umount lustre-MDT0001 complete [ 1614.303168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1614.352779] Lustre: server umount lustre-OST0000 complete [ 1621.667770] Lustre: server umount lustre-OST0001 complete [ 1623.745020] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 1626.905302] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1639.077376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.172743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1639.184472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1639.219441] Lustre: lustre-MDT0000: new disk, initializing [ 1639.246640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1640.616806] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1643.624708] Lustre: server umount lustre-MDT0000 complete [ 1646.945129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.062846] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1647.067022] Lustre: 116462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1647.073112] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1648.433503] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1650.137850] Lustre: server umount lustre-MDT0000 complete [ 1652.965556] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 12:19:32 (1777911572) [ 1653.475890] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1654.045797] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 12:19:33 (1777911573) [ 1659.410921] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1662.846682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.995100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.997189] Lustre: Skipped 5 previous similar messages [ 1664.284940] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1667.080674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.120741] Lustre: 118476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1667.144529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1667.147062] Lustre: Skipped 1 previous similar message [ 1667.183168] Lustre: lustre-MDT0001: new disk, initializing [ 1667.225751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1667.230425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1668.467630] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1670.644283] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1672.777677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1672.866477] Lustre: lustre-OST0000: new disk, initializing [ 1672.868148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1674.164089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1674.168293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1674.197265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1674.660348] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1678.658383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1680.540131] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1680.701611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1680.704667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1680.733874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1684.876728] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1686.038795] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1799.881834] Lustre: 127225:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1799.885392] Lustre: 127225:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1799.888208] Lustre: 127225:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1799.891940] Lustre: 127225:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1799.896079] Lustre: 127225:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1799.900283] Lustre: 127225:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1873.075553] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:23:12 (1777911792) [ 1885.667029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.667896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.670171] Lustre: Skipped 6 previous similar messages [ 1885.672645] Lustre: Skipped 9 previous similar messages [ 1890.741373] Lustre: server umount lustre-MDT0000 complete [ 1890.785820] LustreError: 121664:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.790946] LustreError: 121664:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 1891.863053] LustreError: 119198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911811 with bad export cookie 1735866839623325262 [ 1891.865897] LustreError: 119198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.866334] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1906.143153] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1906.269363] Lustre: server umount lustre-MDT0001 complete [ 1913.660330] Lustre: server umount lustre-OST0000 complete [ 1929.183467] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1929.245615] Lustre: server umount lustre-OST0001 complete [ 1931.188442] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 1934.028509] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1944.811337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.901904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1944.911734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1944.913342] Lustre: Skipped 1 previous similar message [ 1944.947380] Lustre: lustre-MDT0000: new disk, initializing [ 1944.948557] Lustre: Skipped 1 previous similar message [ 1944.973272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.975321] Lustre: Skipped 3 previous similar messages [ 1944.980270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1946.193757] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1950.186675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.223954] Lustre: 130208:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1950.321610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1951.530865] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1954.459735] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1955.606463] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.686805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.565781] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1959.734197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1959.771750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1962.521467] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1965.422489] Lustre: lustre-MDD0000: changelog on [ 1966.230375] Lustre: lustre-MDD0001: changelog on [ 2291.649497] Lustre: 130244:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2318.138288] Lustre: lustre-MDD0001: changelog off [ 2319.247217] Lustre: lustre-MDD0000: changelog off [ 2323.429066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.429105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.430160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.430166] Lustre: Skipped 8 previous similar messages [ 2323.431609] LustreError: Skipped 1 previous similar message [ 2323.434912] Lustre: Skipped 5 previous similar messages [ 2335.199164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2335.262316] Lustre: server umount lustre-OST0000 complete [ 2342.796385] Lustre: server umount lustre-MDT0000 complete [ 2343.904090] LustreError: 131000:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.910299] LustreError: 131000:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 2344.053259] LustreError: 131947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912264 with bad export cookie 1735866839624125712 [ 2344.057455] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.059272] LustreError: 131947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2344.188846] Lustre: server umount lustre-MDT0001 complete [ 2347.743546] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2348.655831] Key type lgssc unregistered [ 2348.766965] LNet: 140535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.769647] LNetError: 140535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.778287] LNet: Removed LNI 192.168.202.122@tcp [ 2349.059093] Key type .llcrypt unregistered [ 2349.060766] Key type ._llcrypt unregistered [ 2354.843711] Key type ._llcrypt registered [ 2354.844789] Key type .llcrypt registered [ 2354.874832] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 2360.726923] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2361.023337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.061292] alg: No test for adler32 (adler32-zlib) [ 2361.915376] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2361.993921] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2363.567137] Key type lgssc registered [ 2363.888550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2374.218778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2374.226702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.315891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2375.326632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2375.365730] Lustre: lustre-MDT0000: new disk, initializing [ 2375.390275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.396437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2376.662486] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2380.710776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.751294] Lustre: 143692:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2380.772560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2380.774102] Lustre: Skipped 1 previous similar message [ 2380.813407] Lustre: lustre-MDT0001: new disk, initializing [ 2380.844903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.855894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2380.859067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2382.169379] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2385.289954] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2386.463643] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2388.810773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2388.915695] Lustre: lustre-OST0000: new disk, initializing [ 2388.918077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2388.948361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.796806] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2391.669996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2391.672913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2391.703192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2393.864072] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2396.055869] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_module obdecho/obdecho [ 2396.715347] LustreError: 146304:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2396.717889] LustreError: 146304:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2397.330425] LustreError: 146401:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2401.248073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2401.252036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.258192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.273682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.275447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.277200] Lustre: Skipped 1 previous similar message [ 2402.282810] Lustre: Skipped 1 previous similar message [ 2406.746508] Lustre: server umount lustre-MDT0000 complete [ 2407.987250] LustreError: 143682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912328 with bad export cookie 6014533544933992945 [ 2407.990552] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.991506] LustreError: 143682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2408.104437] Lustre: server umount lustre-MDT0001 complete [ 2419.663888] Lustre: server umount lustre-OST0000 complete [ 2425.656619] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 2428.370675] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2439.077188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.160817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2439.169465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2439.201797] Lustre: lustre-MDT0000: new disk, initializing [ 2439.224248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.229397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2440.438185] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2444.204384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.236925] Lustre: 150075:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2444.258730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2444.260135] Lustre: Skipped 1 previous similar message [ 2444.290416] Lustre: lustre-MDT0001: new disk, initializing [ 2444.314842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2444.323829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2444.328308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2445.434838] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2448.417216] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2449.605720] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2451.851270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2453.494550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2453.499413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2453.534584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2453.870045] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2456.904221] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2458.246875] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2458.324951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2459.629594] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2459.705411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2463.713174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2463.714156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.715621] LustreError: Skipped 1 previous similar message [ 2463.716270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.721021] Lustre: Skipped 1 previous similar message [ 2463.727555] Lustre: Skipped 2 previous similar messages [ 2466.448150] Lustre: server umount lustre-OST0000 complete [ 2468.832540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.833362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.837110] Lustre: Skipped 1 previous similar message [ 2473.952196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.954641] Lustre: Skipped 2 previous similar messages [ 2482.143112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2482.211979] Lustre: server umount lustre-MDT0000 complete [ 2483.168182] LustreError: 150085:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.175425] LustreError: 150085:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 2483.463081] LustreError: 150067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912403 with bad export cookie 6014533544933993890 [ 2483.466637] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.467105] LustreError: 150067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2483.579674] Lustre: server umount lustre-MDT0001 complete [ 2484.742619] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:33:24 (1777912404) [ 2485.288079] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2485.849817] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:33:25 (1777912405) [ 2488.347045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.513234] LustreError: 154076:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.536192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.539367] Lustre: Skipped 1 previous similar message [ 2489.871975] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2492.689783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.184425] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2495.962799] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.207180] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2499.474337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2501.491808] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2503.328606] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2523.615911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2523.618551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.623523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2523.625286] Lustre: Skipped 2 previous similar messages [ 2527.254150] Lustre: server umount lustre-MDT0000 complete [ 2528.488144] LustreError: 154058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912448 with bad export cookie 6014533544933994758 [ 2528.491719] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.492575] LustreError: 154058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2529.382217] LustreError: 155589:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.280193] LustreError: 155589:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.285101] LustreError: 155589:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2534.654032] Lustre: server umount lustre-MDT0001 complete [ 2536.688687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2540.820811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.905561] LustreError: 157599:0:(llog.c:1646:llog_backup()) MGC192.168.202.122@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2540.908152] Lustre: 157599:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.122@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2558.943922] LustreError: 157604:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.948442] LustreError: 157604:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2568.159597] LustreError: 157606:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.164017] LustreError: 157606:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2586.593293] LustreError: 157604:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.598169] LustreError: 157604:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 14 previous similar messages [ 2601.567270] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2601.568931] Lustre: 157599:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2619.359813] LustreError: 157625:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.363870] LustreError: 157625:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 20 previous similar messages [ 2642.406215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2642.408415] Lustre: Skipped 2 previous similar messages [ 2643.708345] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2646.615314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.522817] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2648.543650] LustreError: 141917:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a216728f x1864276149109888/t0(0) o250->MGC192.168.202.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2649.827385] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2651.484182] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2652.997694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 2653.670902] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2654.082871] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2658.686420] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2663.905158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.905329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2663.906437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.906445] Lustre: Skipped 8 previous similar messages [ 2663.910143] Lustre: Skipped 5 previous similar messages [ 2665.238789] Lustre: server umount lustre-MDT0000 complete [ 2666.468183] LustreError: 154102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912586 with bad export cookie 6014533544934001198 [ 2666.470890] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.471706] LustreError: 154102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2666.592573] Lustre: server umount lustre-MDT0001 complete [ 2678.036982] Lustre: server umount lustre-OST0000 complete [ 2684.436170] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 2687.592389] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2699.413728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.502377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2699.512419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2699.515294] Lustre: Skipped 1 previous similar message [ 2699.552858] Lustre: lustre-MDT0000: new disk, initializing [ 2699.554027] Lustre: Skipped 1 previous similar message [ 2699.577252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.579451] Lustre: Skipped 1 previous similar message [ 2699.585070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2700.843521] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2704.898974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.936653] Lustre: 162409:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2705.030469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2706.277578] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2709.244898] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2710.389310] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2712.590614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.520675] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2717.539797] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2718.807816] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2721.783302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2721.816776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2721.931820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2723.210702] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2723.281686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2726.882472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.882765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2726.886622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2726.889954] Lustre: Skipped 3 previous similar messages [ 2729.997103] Lustre: server umount lustre-OST0000 complete [ 2737.308349] Lustre: server umount lustre-MDT0000 complete [ 2738.524030] LustreError: 162402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912658 with bad export cookie 6014533544934017998 [ 2738.526642] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2738.528377] LustreError: 162402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2738.650370] Lustre: server umount lustre-MDT0001 complete [ 2741.531401] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:37:40 (1777912660) [ 2743.754343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.901814] LustreError: 166431:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.907024] LustreError: 166431:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 23 previous similar messages [ 2745.145275] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2747.834629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.232882] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2750.911689] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2752.097269] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2754.365832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.468197] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2758.225500] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2777.312687] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:38:16 (1777912696) [ 2780.129996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.130463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2780.133922] Lustre: Skipped 4 previous similar messages [ 2780.135842] LustreError: Skipped 1 previous similar message [ 2784.331022] Lustre: server umount lustre-OST0000 complete [ 2791.703171] Lustre: server umount lustre-MDT0000 complete [ 2792.939986] LustreError: 168799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912712 with bad export cookie 6014533544934018866 [ 2792.943362] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2792.945102] LustreError: 168799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2793.056862] Lustre: server umount lustre-MDT0001 complete [ 2796.608215] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2797.614611] Key type lgssc unregistered [ 2797.735409] LNet: 169822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.737876] LNetError: 169822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.747341] LNet: Removed LNI 192.168.202.122@tcp [ 2798.050979] Key type .llcrypt unregistered [ 2798.052038] Key type ._llcrypt unregistered [ 2805.436555] Key type ._llcrypt registered [ 2805.438197] Key type .llcrypt registered [ 2805.475420] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2805.808059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.873453] alg: No test for adler32 (adler32-zlib) [ 2806.728814] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2806.807362] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2808.391113] Key type lgssc registered [ 2808.762462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2811.430160] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2811.436286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2826.847275] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2831.967171] LustreError: 171025:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 2837.087388] LustreError: 171025:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2837.090749] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2837.092614] LustreError: MGC192.168.202.122@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2837.096889] LustreError: 171025:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2837.099885] LustreError: 171025:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 2837.102546] LustreError: 171025:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2837.124626] Lustre: server umount lustre-OST0000 complete [ 2837.125919] LustreError: 171025:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2839.908690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.058949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.265579] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2844.992602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.121332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.310305] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2847.930612] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2848.987742] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2851.048803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.115957] LustreError: 173111:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2851.120237] Lustre: 173111:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.202.122@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2851.161105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.912165] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2854.603798] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2855.391817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2855.395245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.400915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.512540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.514132] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.517990] Lustre: Skipped 1 previous similar message [ 2861.464073] Lustre: server umount lustre-MDT0000 complete [ 2862.645357] LustreError: 171505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912782 with bad export cookie 9127715911726468942 [ 2862.648036] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.651831] LustreError: 171505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2862.760089] Lustre: server umount lustre-MDT0001 complete [ 2874.269015] Lustre: server umount lustre-OST0000 complete [ 2877.183878] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 2880.732294] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2892.163384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.251798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2892.262593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2892.301912] Lustre: lustre-MDT0000: new disk, initializing [ 2892.323848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.330394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2893.592558] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2897.729239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.773599] Lustre: 176483:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2897.796690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2897.798337] Lustre: Skipped 1 previous similar message [ 2897.835545] Lustre: lustre-MDT0001: new disk, initializing [ 2897.865616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2897.877292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2897.880686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2899.243094] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2902.250278] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2903.433546] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2905.618176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2905.700424] Lustre: lustre-OST0000: new disk, initializing [ 2905.701805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2907.466724] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2910.659912] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2911.731675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2911.734490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2911.765591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2912.023420] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2912.092095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2913.369113] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2913.443981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2916.833942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2916.834571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.837024] LustreError: Skipped 1 previous similar message [ 2916.841500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.844423] Lustre: Skipped 3 previous similar messages [ 2920.140326] Lustre: server umount lustre-OST0000 complete [ 2921.952220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.952838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.956169] Lustre: Skipped 2 previous similar messages [ 2927.072232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2927.074462] Lustre: Skipped 2 previous similar messages [ 2927.448584] Lustre: server umount lustre-MDT0000 complete [ 2928.619068] LustreError: 176477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912848 with bad export cookie 9127715911726469635 [ 2928.622028] LustreError: 176477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2928.622307] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.733488] Lustre: server umount lustre-MDT0001 complete [ 2931.525483] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:40:51 (1777912851) [ 2937.312910] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2938.393291] Key type lgssc unregistered [ 2938.531403] LNet: 181002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.534627] LNetError: 181002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2938.542374] LNet: Removed LNI 192.168.202.122@tcp [ 2938.846719] Key type .llcrypt unregistered [ 2938.848508] Key type ._llcrypt unregistered [ 2945.952412] Key type ._llcrypt registered [ 2945.954014] Key type .llcrypt registered [ 2945.993871] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2946.293820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2946.300783] alg: No test for adler32 (adler32-zlib) [ 2947.150154] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2947.230673] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2948.807202] Key type lgssc registered [ 2949.130294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2951.553992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2951.560288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.713546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.938639] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2956.633976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.750842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2958.012907] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2959.709030] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2960.922980] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2963.429770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.504744] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2963.505990] LustreError: 183817:0:(llog.c:1687:llog_backup()) MGC192.168.202.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2963.508451] Lustre: 183817:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.202.122@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2963.545085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.408636] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2967.290187] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2972.131829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2972.131900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.134171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.144523] Lustre: Skipped 2 previous similar messages [ 2977.249070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2977.251459] Lustre: Skipped 4 previous similar messages [ 2982.367206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2982.370069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.372033] Lustre: Skipped 2 previous similar messages [ 2982.428724] Lustre: server umount lustre-MDT0000 complete [ 2983.551024] LustreError: 182162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912903 with bad export cookie 14602703349050084212 [ 2983.554023] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.554467] LustreError: 182162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.658982] Lustre: server umount lustre-MDT0001 complete [ 2994.067803] Lustre: server umount lustre-OST0000 complete [ 2997.018117] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3000.394517] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3012.023138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.108104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3012.118607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3012.155745] Lustre: lustre-MDT0000: new disk, initializing [ 3012.178827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.185470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3013.485535] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3017.584996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.624328] Lustre: 187186:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3017.641780] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3017.642965] Lustre: Skipped 1 previous similar message [ 3017.644151] LustreError: 187950:0:(llog.c:1687:llog_backup()) MGC192.168.202.122@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3017.646672] LustreError: 187950:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3017.648427] Lustre: 187950:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.122@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3017.655530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3017.657958] Lustre: Skipped 1 previous similar message [ 3017.696931] Lustre: lustre-MDT0001: new disk, initializing [ 3017.730304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3017.741339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3017.744705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3019.044199] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3022.057517] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3023.204484] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3025.400661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.510813] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3025.512475] Lustre: Skipped 1 previous similar message [ 3025.514210] LustreError: 188904:0:(llog.c:1687:llog_backup()) MGC192.168.202.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3025.518365] LustreError: 188904:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3025.521178] Lustre: 188904:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.122@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3025.525431] Lustre: 188904:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3025.535790] Lustre: lustre-OST0000: new disk, initializing [ 3025.537922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3025.677356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3025.682687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3025.715038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3027.432378] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3030.450717] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3031.834923] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3031.901128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3033.200601] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3033.275431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3036.130580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.130600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.139550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.142552] Lustre: Skipped 1 previous similar message [ 3040.012000] Lustre: server umount lustre-OST0000 complete [ 3041.248190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.251581] Lustre: Skipped 1 previous similar message [ 3041.253209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.254962] Lustre: Skipped 1 previous similar message [ 3047.315027] Lustre: server umount lustre-MDT0000 complete [ 3048.554493] LustreError: 187178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912968 with bad export cookie 14602703349050084898 [ 3048.557874] LustreError: 187178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3048.557966] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.688417] Lustre: server umount lustre-MDT0001 complete [ 3051.345693] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:42:50 (1777912970) [ 3056.583343] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3059.706376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.841164] LustreError: 191884:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.847918] LustreError: 191884:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3059.857115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.859075] Lustre: Skipped 1 previous similar message [ 3061.069861] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3063.696457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.034868] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3065.951365] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3068.151645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.128731] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3072.888317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3072.948169] Lustre: lustre-OST0001: new disk, initializing [ 3072.950253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3074.772153] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3078.141570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3078.144318] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3078.173323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3080.065407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3085.629991] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3086.223221] Lustre: DEBUG MARKER: ADD OST3 [ 3087.964425] loop: module loaded [ 3087.980985] loop0: detected capacity change from 0 to 204800000 [ 3088.060969] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3088.064082] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3088.067340] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3090.143154] loop0: detected capacity change from 0 to 204800000 [ 3092.022901] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.105340] Lustre: lustre-OST0003: new disk, initializing [ 3092.108121] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3092.159834] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3092.161922] Lustre: Skipped 3 previous similar messages [ 3093.084536] Lustre: DEBUG MARKER: STOP OST3 [ 3093.740888] Lustre: Failing over lustre-OST0003 [ 3093.790084] Lustre: server umount lustre-OST0003 complete [ 3093.944172] LustreError: 193333:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.382638] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3098.379323] Lustre: DEBUG MARKER: START OST3 again [ 3099.116178] LustreError: 193516:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.121421] LustreError: 193516:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 3100.685017] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.662800] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3105.769917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3105.772612] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3105.783902] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3116.200092] Lustre: Failing over lustre-OST0003 [ 3116.240752] Lustre: server umount lustre-OST0003 complete [ 3119.013671] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.019570] Lustre: Skipped 1 previous similar message [ 3119.022285] LustreError: 193335:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.027126] LustreError: 193335:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3124.195709] LustreError: 193333:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.200159] LustreError: 193333:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3129.314136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.316574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.318522] Lustre: Skipped 3 previous similar messages [ 3129.322725] Lustre: Skipped 5 previous similar messages [ 3134.436670] LustreError: 193334:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.444241] LustreError: 193334:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3135.010235] Lustre: server umount lustre-MDT0000 complete [ 3136.209169] LustreError: 194889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913056 with bad export cookie 14602703349050085766 [ 3136.212565] LustreError: 194889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3136.213689] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.552851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.555879] Lustre: Skipped 1 previous similar message [ 3142.450445] Lustre: server umount lustre-MDT0001 complete [ 3143.717974] Lustre: server umount lustre-OST0000 complete [ 3144.935429] Lustre: server umount lustre-OST0001 complete [ 3146.907479] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3149.369481] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3161.022496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.110285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3161.120581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3161.163401] Lustre: lustre-MDT0000: new disk, initializing [ 3161.189115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.192025] Lustre: Skipped 1 previous similar message [ 3161.198042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3162.473901] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3166.433143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.473858] Lustre: 201192:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3166.575037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3167.925678] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3170.895822] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3172.015649] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3174.112676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.683356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3175.725487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3176.024830] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3179.027892] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3180.380647] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3180.455924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3181.733373] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3181.804541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3186.146322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.148108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.149758] Lustre: Skipped 1 previous similar message [ 3186.150482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.152188] LustreError: Skipped 1 previous similar message [ 3186.156343] Lustre: Skipped 9 previous similar messages [ 3188.563060] Lustre: server umount lustre-OST0000 complete [ 3196.000097] Lustre: server umount lustre-MDT0000 complete [ 3196.383799] LustreError: 201203:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.389078] LustreError: 201203:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 3197.264078] LustreError: 201184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913117 with bad export cookie 14602703349050087999 [ 3197.266678] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.267277] LustreError: 201184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3197.386248] Lustre: server umount lustre-MDT0001 complete [ 3198.525690] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:45:18 (1777913118) [ 3203.406546] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3206.789253] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3218.167549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.272330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3218.286141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3218.287865] Lustre: Skipped 3 previous similar messages [ 3218.326188] Lustre: lustre-MDT0000: new disk, initializing [ 3218.327326] Lustre: Skipped 2 previous similar messages [ 3218.355745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3218.358051] Lustre: Skipped 2 previous similar messages [ 3219.573310] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3223.663380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.702063] Lustre: 207354:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3223.813419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3225.067802] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3228.126286] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3229.287667] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3231.513711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.632285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.634255] Lustre: Skipped 4 previous similar messages [ 3233.371098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3233.371464] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3236.345690] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3237.830919] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3237.896410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3239.150894] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3239.219096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3243.492835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3243.493816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.494588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3243.494594] Lustre: Skipped 2 previous similar messages [ 3243.496878] LustreError: Skipped 2 previous similar messages [ 3243.506107] Lustre: Skipped 4 previous similar messages [ 3254.239069] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3254.294129] Lustre: server umount lustre-OST0000 complete [ 3261.611479] Lustre: server umount lustre-MDT0000 complete [ 3263.339177] LustreError: 208133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913183 with bad export cookie 14602703349050088881 [ 3263.341462] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.342066] LustreError: 208133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3263.462640] Lustre: server umount lustre-MDT0001 complete [ 3266.923546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.082286] LustreError: 211168:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.432582] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3272.079421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.660663] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3275.887991] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.336273] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3279.793815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.473235] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3284.239772] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3302.881188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3307.746119] Lustre: server umount lustre-OST0000 complete [ 3313.119777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3313.121635] LustreError: Skipped 1 previous similar message [ 3313.122720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.125631] Lustre: Skipped 4 previous similar messages [ 3313.127082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.128557] Lustre: Skipped 11 previous similar messages [ 3315.409959] Lustre: server umount lustre-MDT0000 complete [ 3316.614288] LustreError: 211149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913236 with bad export cookie 14602703349050089763 [ 3316.617471] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.618135] LustreError: 211149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3316.741459] Lustre: server umount lustre-MDT0001 complete [ 3419.772143] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3420.877025] Key type lgssc unregistered [ 3421.014291] LNet: 214496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.016793] LNetError: 214496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.022431] LNet: Removed LNI 192.168.202.122@tcp [ 3421.325119] Key type .llcrypt unregistered [ 3421.326287] Key type ._llcrypt unregistered [ 3430.201406] Key type ._llcrypt registered [ 3430.202528] Key type .llcrypt registered [ 3430.240042] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3436.314159] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3436.666344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3436.689636] alg: No test for adler32 (adler32-zlib) [ 3437.542702] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3437.628714] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3439.207182] Key type lgssc registered [ 3439.697944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3453.890768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3453.900929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.023926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3455.036233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3455.077976] Lustre: lustre-MDT0000: new disk, initializing [ 3455.106358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.113118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3456.404846] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3461.187903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.244624] Lustre: 217785:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3461.279385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3461.283140] Lustre: Skipped 1 previous similar message [ 3461.325694] Lustre: lustre-MDT0001: new disk, initializing [ 3461.361377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3461.379140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3461.383180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3462.896683] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3466.097328] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3467.246547] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3469.773237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.897610] Lustre: lustre-OST0000: new disk, initializing [ 3469.899380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3469.928344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.857687] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3474.910744] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3476.245623] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3479.542125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3479.545134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3479.575713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3480.402711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3481.691949] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3481.768436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3484.643390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3484.644746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.647428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3484.650301] Lustre: Skipped 1 previous similar message [ 3488.476986] Lustre: server umount lustre-OST0000 complete [ 3489.760278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.760753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.763716] Lustre: Skipped 1 previous similar message [ 3489.765383] Lustre: Skipped 1 previous similar message [ 3494.880159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3494.881875] Lustre: Skipped 2 previous similar messages [ 3495.830783] Lustre: server umount lustre-MDT0000 complete [ 3496.909380] LustreError: 217778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913416 with bad export cookie 17871595151126397609 [ 3496.912276] LustreError: 217778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3496.912510] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.015461] Lustre: server umount lustre-MDT0001 complete [ 3502.269595] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3505.647841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.806479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.011651] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3510.380298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.498913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3511.766656] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3512.711815] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3515.428291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.449638] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3520.315260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.384506] Lustre: lustre-OST0001: new disk, initializing [ 3520.386926] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3520.429560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.432393] Lustre: Skipped 1 previous similar message [ 3522.450908] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3527.167748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3527.171664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3527.202055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3528.010480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3533.971839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3537.736483] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:50:57 (1777913457) [ 3541.472371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3541.476702] LustreError: Skipped 1 previous similar message [ 3541.478604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.484148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.598333] Lustre: server umount lustre-MDT0000 complete [ 3545.773158] LustreError: 223419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913465 with bad export cookie 17871595151126398477 [ 3545.777213] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3545.777626] LustreError: 223419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3545.879260] Lustre: server umount lustre-MDT0001 complete [ 3557.836949] Lustre: server umount lustre-OST0000 complete [ 3569.506155] Lustre: server umount lustre-OST0001 complete [ 3571.633156] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3574.485355] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3587.771203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.864299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3587.874636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3587.909264] Lustre: lustre-MDT0000: new disk, initializing [ 3587.934075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.940688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3589.291513] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3594.005118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.052096] Lustre: 228462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3594.186509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3595.608467] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3599.259773] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.523452] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3603.571920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.730113] Lustre: lustre-OST0000: new disk, initializing [ 3603.732382] Lustre: Skipped 1 previous similar message [ 3603.734628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3603.737300] Lustre: Skipped 2 previous similar messages [ 3604.861531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3604.864311] Lustre: Skipped 1 previous similar message [ 3604.866867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3604.912354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3605.994998] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3609.563636] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3611.033531] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3611.122746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3612.637037] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3612.737038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3615.203376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.204114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.207348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.207355] Lustre: Skipped 4 previous similar messages [ 3615.207870] LustreError: Skipped 1 previous similar message [ 3615.228776] Lustre: Skipped 4 previous similar messages [ 3619.748837] Lustre: server umount lustre-OST0000 complete [ 3624.928770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3624.935517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.944221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.948102] Lustre: Skipped 1 previous similar message [ 3627.520728] Lustre: server umount lustre-MDT0000 complete [ 3629.393720] LustreError: 228455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913549 with bad export cookie 17871595151126400598 [ 3629.398121] LustreError: 228455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3629.398722] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3629.560996] Lustre: server umount lustre-MDT0001 complete [ 3633.118974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.290148] LustreError: 232272:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.296655] LustreError: 232272:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3633.314999] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.317022] Lustre: Skipped 2 previous similar messages [ 3634.641355] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3638.085132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.676800] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3641.500624] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3642.882836] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3645.424495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3647.343254] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3649.014530] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3797.474834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3797.475933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3797.476445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3797.476449] Lustre: Skipped 2 previous similar messages [ 3797.478312] Lustre: Skipped 2 previous similar messages [ 3797.484563] LustreError: Skipped 1 previous similar message [ 3800.146157] Lustre: server umount lustre-OST0000 complete [ 3807.838685] Lustre: server umount lustre-MDT0000 complete [ 3809.143341] LustreError: 232253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913729 with bad export cookie 17871595151126401480 [ 3809.146797] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.147699] LustreError: 232253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3809.270101] Lustre: server umount lustre-MDT0001 complete [ 3910.977556] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3911.950700] Key type lgssc unregistered [ 3912.075400] LNet: 235523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.077720] LNetError: 235523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.087284] LNet: Removed LNI 192.168.202.122@tcp [ 3912.393156] Key type .llcrypt unregistered [ 3912.394547] Key type ._llcrypt unregistered [ 3921.097997] Key type ._llcrypt registered [ 3921.098937] Key type .llcrypt registered [ 3921.138093] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3927.331113] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3927.757820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3927.806154] alg: No test for adler32 (adler32-zlib) [ 3928.677075] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3928.782095] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3930.367112] Key type lgssc registered [ 3930.866840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3947.406638] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3951.523327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3951.529864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.643822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3952.656725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3952.696214] Lustre: lustre-MDT0000: new disk, initializing [ 3952.723651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.729791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3954.434602] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3959.406814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.451775] Lustre: 239511:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3959.476884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3959.479484] Lustre: Skipped 1 previous similar message [ 3959.524557] Lustre: lustre-MDT0001: new disk, initializing [ 3959.565196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3959.578103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3959.584304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3961.098353] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3963.678387] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3965.985951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.109414] Lustre: lustre-OST0000: new disk, initializing [ 3966.111079] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3966.146847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3968.069526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3968.076305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3968.124678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3968.509963] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3973.409830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3973.516060] Lustre: lustre-OST0001: new disk, initializing [ 3973.518697] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3973.568253] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3975.886808] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3976.352337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3976.358580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3976.404053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3981.936945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3988.087760] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3995.616298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3995.618668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.622646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.154651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.157360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.166597] Lustre: Skipped 1 previous similar message [ 3998.344154] Lustre: server umount lustre-MDT0000 complete [ 4000.085108] LustreError: 241962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913920 with bad export cookie 3502292962649002179 [ 4000.091079] LustreError: 241962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4000.091351] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.232479] Lustre: server umount lustre-MDT0001 complete [ 4012.489231] Lustre: server umount lustre-OST0000 complete [ 4024.519056] Lustre: server umount lustre-OST0001 complete [ 4030.593115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4035.133048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4038.758845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4043.091540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4050.680297] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4054.938021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.944404] Lustre: lustre-MDT0000: reset Object Index mappings [ 4055.127400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.761257] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4060.129340] LustreError: 246930:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.143734] LustreError: 246930:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4060.524118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.531112] Lustre: lustre-MDT0001: reset Object Index mappings [ 4062.330502] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4063.797636] Lustre: 248035:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4067.023535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4067.192139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.196046] Lustre: Skipped 1 previous similar message [ 4069.465596] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4070.248317] LustreError: 248392:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.580232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.335784] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4081.148863] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4081.149125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4086.005214] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4087.592770] Lustre: 249873:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4091.361686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.364157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.369204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.374352] Lustre: Skipped 4 previous similar messages [ 4091.379315] Lustre: Skipped 5 previous similar messages [ 4095.487263] Lustre: server umount lustre-MDT0000 complete [ 4096.481075] LustreError: 246930:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.485145] LustreError: 246930:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4096.968963] LustreError: 249875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914017 with bad export cookie 3502292962649008962 [ 4096.974585] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4096.977204] LustreError: 249875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4097.137642] Lustre: server umount lustre-MDT0001 complete [ 4108.822799] Lustre: server umount lustre-OST0000 complete [ 4120.860281] Lustre: server umount lustre-OST0001 complete [ 4130.389990] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:00:49 (1777914049) [ 4137.117371] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 4141.078406] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4157.124181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.234452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4157.248316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4157.287525] Lustre: lustre-MDT0000: new disk, initializing [ 4157.315212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.318781] Lustre: Skipped 1 previous similar message [ 4157.324790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4158.837896] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4164.685173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.737721] Lustre: 254100:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4164.882251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4166.556782] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4170.696657] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4172.537742] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4175.373410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4175.548792] Lustre: lustre-OST0000: new disk, initializing [ 4175.550516] Lustre: Skipped 1 previous similar message [ 4175.552276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4175.554298] Lustre: Skipped 2 previous similar messages [ 4177.156530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4177.161106] Lustre: Skipped 1 previous similar message [ 4177.164083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4177.197387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4177.975460] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4181.433807] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4182.787216] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4182.856100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4184.185427] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4184.255143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4187.620768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.621398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4187.625316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4187.626361] LustreError: Skipped 1 previous similar message [ 4187.628019] Lustre: Skipped 2 previous similar messages [ 4192.739996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4192.741809] Lustre: Skipped 1 previous similar message [ 4199.391083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4199.443085] Lustre: server umount lustre-OST0000 complete [ 4200.928356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4200.930510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.933659] Lustre: Skipped 1 previous similar message [ 4200.935318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.936974] Lustre: Skipped 2 previous similar messages [ 4215.263228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4215.345514] Lustre: server umount lustre-MDT0000 complete [ 4217.109660] LustreError: 254888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914137 with bad export cookie 3502292962649011034 [ 4217.114287] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.117299] LustreError: 254888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4217.292262] Lustre: server umount lustre-MDT0001 complete [ 4224.203802] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4228.833773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4229.061231] LustreError: 258609:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.085052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.089508] Lustre: Skipped 2 previous similar messages [ 4230.629393] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4234.210842] LustreError: 258610:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.719757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.657991] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4238.046925] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4241.336585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.428361] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4248.660288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4248.758380] Lustre: lustre-OST0001: new disk, initializing [ 4248.761824] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4250.525201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4250.528722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4250.567032] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4251.416444] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4256.633393] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4258.282380] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4260.622905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4260.634759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4264.933095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.940117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.941306] Lustre: Skipped 3 previous similar messages [ 4264.945505] Lustre: Skipped 8 previous similar messages [ 4269.245872] Lustre: server umount lustre-MDT0000 complete [ 4270.049952] LustreError: 259939:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.063845] LustreError: 259939:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4270.631142] LustreError: 259709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914190 with bad export cookie 3502292962649011916 [ 4270.635399] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4270.637184] LustreError: 259709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4270.761942] Lustre: server umount lustre-MDT0001 complete [ 4282.399968] Lustre: server umount lustre-OST0000 complete [ 4293.199140] Lustre: server umount lustre-OST0001 complete [ 4300.409212] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4304.628094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.834181] LustreError: 263881:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.866350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.870805] Lustre: Skipped 3 previous similar messages [ 4306.765938] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4309.772487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.252415] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4312.704542] Lustre: 264987:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4315.781535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.534531] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4322.077057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.062649] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4329.466820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4329.480426] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4333.604234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4335.379415] Lustre: 266830:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4337.886765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4337.893618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4341.854451] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:04:21 (1777914261) [ 4355.045394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.049849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4355.055478] Lustre: Skipped 3 previous similar messages [ 4355.059375] Lustre: Skipped 2 previous similar messages [ 4359.865405] Lustre: server umount lustre-MDT0000 complete [ 4360.164523] LustreError: 263882:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.174836] LustreError: 263882:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4361.653382] LustreError: 263861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914281 with bad export cookie 3502292962649018223 [ 4361.660916] LustreError: 263861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4361.661503] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.901790] Lustre: server umount lustre-MDT0001 complete [ 4384.223106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4384.267958] Lustre: server umount lustre-OST0000 complete [ 4391.721256] Lustre: server umount lustre-OST0001 complete [ 4394.075471] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 4398.165665] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4418.458402] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4422.314036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.466060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4422.482725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4422.538510] Lustre: lustre-MDT0000: new disk, initializing [ 4422.583317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4424.428395] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4430.251582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.311769] Lustre: 271010:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4430.458516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4432.302069] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4438.181411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.389727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.392551] Lustre: Skipped 5 previous similar messages [ 4440.222397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4441.126445] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4446.696612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.344119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4449.300132] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4454.809774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4456.468614] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4456.472093] Lustre: Skipped 1 previous similar message [ 4462.704747] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:06:21 (1777914381) [ 4466.145264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4466.150325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.158160] Lustre: Skipped 2 previous similar messages [ 4466.161760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4466.165640] Lustre: Skipped 5 previous similar messages [ 4469.955840] Lustre: server umount lustre-MDT0000 complete [ 4471.708882] LustreError: 271003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914391 with bad export cookie 3502292962649026105 [ 4471.714522] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.716401] LustreError: 271003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4471.863802] Lustre: server umount lustre-MDT0001 complete [ 4483.027164] Lustre: server umount lustre-OST0000 complete [ 4494.140488] Lustre: server umount lustre-OST0001 complete [ 4498.436499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.679240] LustreError: 275693:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.690536] LustreError: 275693:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4500.550308] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4504.655220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.825742] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4509.594446] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4511.500552] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4514.980612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.827201] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4520.160489] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4536.724282] loop1: detected capacity change from 0 to 204800000 [ 4536.863154] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4536.867476] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4536.871823] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4540.076504] loop1: detected capacity change from 0 to 204800000 [ 4543.456546] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.520981] LustreError: 275681:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs5344: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4543.527817] LustreError: 275681:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs5344 [ 4543.530786] LustreError: 275681:0:(mgs_handler.c:517:mgs_target_reg()) fs5344-MDT0001: Error checking target index: rc = -13 [ 4543.535594] LustreError: fs5344-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4543.540888] LustreError: 279138:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 4543.543534] LustreError: 279138:0:(tgt_mount.c:2084:server_put_super()) no obd fs5344-MDT0001 [ 4543.546748] LustreError: 279138:0:(tgt_mount.c:116:server_deregister_mount()) fs5344-MDT0001 not registered [ 4543.569787] Lustre: server umount fs5344-MDT0001 complete [ 4543.571518] LustreError: 279138:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4549.135118] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.197907] Lustre: Setting parameter fs5344-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5344-MDT0001 [ 4549.260053] Lustre: srv-fs5344-MDT0001: No data found on store. Initialize space. [ 4549.268962] Lustre: Skipped 4 previous similar messages [ 4549.359379] Lustre: fs5344-MDT0001: new disk, initializing [ 4549.364749] Lustre: Skipped 3 previous similar messages [ 4549.440617] Lustre: srv-fs5344-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4549.445509] Lustre: 279604:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5344-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4549.450769] LustreError: 279604:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs5344-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4552.727780] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4557.198902] Lustre: server umount fs5344-MDT0001 complete [ 4561.762039] loop1: detected capacity change from 0 to 204800000 [ 4563.606804] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.727424] Lustre: 281151:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5344-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4563.732906] LustreError: 281151:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs5344-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4564.989531] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4566.551391] Lustre: server umount fs5344-MDT0001 complete [ 4570.594398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.598695] Lustre: Skipped 5 previous similar messages [ 4575.835753] Lustre: server umount lustre-MDT0000 complete [ 4578.844329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.894167] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4580.271643] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4582.593132] loop1: detected capacity change from 0 to 204800000 [ 4584.427824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4584.469986] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.587453] Lustre: 283561:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5344-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4584.591717] LustreError: 283561:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs5344-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4585.849131] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4587.541151] Lustre: server umount fs5344-MDT0001 complete [ 4594.658468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4594.660467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4594.662342] Lustre: Skipped 12 previous similar messages [ 4600.214677] Lustre: server umount lustre-OST0000 complete [ 4607.636312] Lustre: server umount lustre-MDT0000 complete [ 4608.795058] LustreError: 284846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914528 with bad export cookie 3502292962649028786 [ 4608.920183] Lustre: server umount lustre-MDT0001 complete [ 4612.199881] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 4613.169229] Key type lgssc unregistered [ 4613.293320] LNet: 285867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4613.296306] LNetError: 285867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4613.302301] LNet: Removed LNI 192.168.202.122@tcp [ 4613.594138] Key type .llcrypt unregistered [ 4613.595832] Key type ._llcrypt unregistered [ 4620.543267] Key type ._llcrypt registered [ 4620.544218] Key type .llcrypt registered [ 4620.582093] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4620.918419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4620.995531] alg: No test for adler32 (adler32-zlib) [ 4621.844892] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 4621.915115] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 4623.495135] Key type lgssc registered [ 4623.820320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.481185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.488120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.684171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.996147] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4631.747502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.880444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4633.164635] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4634.902171] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.026895] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4638.402411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4638.541545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.588844] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4641.636792] LustreError: 288698:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4641.644504] LustreError: 288698:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4642.399956] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4646.883641] LustreError: 289191:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.002318] LustreError: 289187:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.007303] LustreError: 289187:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4655.013564] LustreError: 288699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.022829] LustreError: 288699:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4659.594807] loop1: detected capacity change from 0 to 204800000 [ 4659.662512] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4659.666259] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4659.670771] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4660.195253] LustreError: 288699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.202601] LustreError: 288699:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4661.457797] loop1: detected capacity change from 0 to 204800000 [ 4663.363861] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.417599] Lustre: fs24422-OST0000: new disk, initializing [ 4663.419327] Lustre: srv-fs24422-OST0000: No data found on store. Initialize space. [ 4663.442251] Lustre: fs24422-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.336824] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4668.519555] LustreError: 289187:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.527822] LustreError: 289187:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4668.691846] Lustre: server umount fs24422-OST0000 complete [ 4672.685679] loop1: detected capacity change from 0 to 204800000 [ 4674.569540] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.634793] Lustre: fs24422-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.511118] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4679.300576] loop2: detected capacity change from 0 to 204800000 [ 4679.369791] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4679.377162] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4679.380595] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4681.118154] loop2: detected capacity change from 0 to 204800000 [ 4682.998040] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4683.030913] LustreError: fs24422-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4683.037967] LustreError: fs24422-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4683.041891] LustreError: 293562:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 4683.044942] LustreError: 293562:0:(tgt_mount.c:2084:server_put_super()) no obd fs24422-OST0001 [ 4683.048194] LustreError: 293562:0:(tgt_mount.c:116:server_deregister_mount()) fs24422-OST0001 not registered [ 4683.061672] Lustre: server umount fs24422-OST0001 complete [ 4683.063214] LustreError: 293562:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4685.795864] LustreError: 288699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.802345] LustreError: 288699:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 10 previous similar messages [ 4686.575109] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.621900] Lustre: fs24422-OST0001: new disk, initializing [ 4686.624073] Lustre: srv-fs24422-OST0001: No data found on store. Initialize space. [ 4686.646370] Lustre: fs24422-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4688.499991] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4692.091918] Lustre: server umount fs24422-OST0000 complete [ 4695.069608] Lustre: server umount fs24422-OST0001 complete [ 4698.843248] loop1: detected capacity change from 0 to 204800000 [ 4700.578863] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.349914] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4704.410719] loop2: detected capacity change from 0 to 204800000 [ 4706.158213] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.220181] Lustre: fs24422-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4706.221786] Lustre: Skipped 1 previous similar message [ 4707.976513] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4710.777223] Lustre: server umount fs24422-OST0000 complete [ 4713.863435] Lustre: server umount fs24422-OST0001 complete [ 4718.560163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.560296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.564275] LustreError: Skipped 1 previous similar message [ 4718.568379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4718.572972] Lustre: Skipped 1 previous similar message [ 4721.633104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.633123] LustreError: 289187:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.639994] LustreError: 289187:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 19 previous similar messages [ 4723.604642] Lustre: server umount lustre-OST0000 complete [ 4726.752783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.753923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.756049] Lustre: Skipped 2 previous similar messages [ 4726.759192] Lustre: Skipped 2 previous similar messages [ 4731.039480] Lustre: server umount lustre-MDT0000 complete [ 4732.294247] LustreError: 291288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914652 with bad export cookie 7137582622326351735 [ 4732.297821] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4732.297923] LustreError: 291288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4732.428142] Lustre: server umount lustre-MDT0001 complete [ 4736.188842] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 4737.196529] Key type lgssc unregistered [ 4737.323337] LNet: 299954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.325632] LNetError: 299954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.333338] LNet: Removed LNI 192.168.202.122@tcp [ 4737.625100] Key type .llcrypt unregistered [ 4737.626139] Key type ._llcrypt unregistered [ 4747.437151] Key type ._llcrypt registered [ 4747.438538] Key type .llcrypt registered [ 4747.484916] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 4754.089875] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4754.418896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4754.425759] alg: No test for adler32 (adler32-zlib) [ 4755.282150] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 4755.363230] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 4756.951153] Key type lgssc registered [ 4757.301536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4769.808724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4769.815637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.903800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4770.915279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4770.958244] Lustre: lustre-MDT0000: new disk, initializing [ 4770.982461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.988357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4772.267311] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4776.628980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.670277] Lustre: 303751:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4776.703597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4776.707738] Lustre: Skipped 1 previous similar message [ 4776.751583] Lustre: lustre-MDT0001: new disk, initializing [ 4776.782578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4776.794534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4776.799872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4778.293751] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4781.515436] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4782.785103] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4784.952583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4785.074768] Lustre: lustre-OST0000: new disk, initializing [ 4785.076493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4785.106308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.951803] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4789.754749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4789.760135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4789.792527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4790.219646] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4790.964675] Lustre: Mounted lustre-client [ 4791.263594] LustreError: 306244:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c128b7d7800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4791.292197] Lustre: Unmounted lustre-client [ 4796.385648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.387289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4796.389394] Lustre: Skipped 1 previous similar message [ 4796.395367] Lustre: Skipped 1 previous similar message [ 4801.295256] Lustre: server umount lustre-OST0000 complete [ 4802.527556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4802.530069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.533809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4806.623996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4806.624542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.665928] Lustre: server umount lustre-MDT0000 complete [ 4809.856949] LustreError: 303745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914729 with bad export cookie 16640732362552672885 [ 4809.859776] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4809.862266] LustreError: 303745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4809.978146] Lustre: server umount lustre-MDT0001 complete [ 4814.079471] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 4815.537582] Key type lgssc unregistered [ 4848.479294] LNet: 307521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4848.481680] LNetError: 307521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4921.189366] LNet: Removed LNI 192.168.202.122@tcp [ 4921.509270] Key type .llcrypt unregistered [ 4921.510605] Key type ._llcrypt unregistered [ 4929.063786] Key type ._llcrypt registered [ 4929.065499] Key type .llcrypt registered [ 4929.117180] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 4937.089506] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4937.413810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.422645] alg: No test for adler32 (adler32-zlib) [ 4938.281567] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 4938.369693] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 4939.959162] Key type lgssc registered [ 4940.565700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4950.088156] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:14:29 (1777914869) [ 4953.083465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4953.089903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.194684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4954.211260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4954.254132] Lustre: lustre-MDT0000: new disk, initializing [ 4954.276404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.282839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4955.429109] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4959.240438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.275322] Lustre: 310686:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.295076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4959.297530] Lustre: Skipped 1 previous similar message [ 4959.334345] Lustre: lustre-MDT0001: new disk, initializing [ 4959.363507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.373952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4959.377118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4960.606667] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4964.218247] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4965.537573] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4968.273599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4968.421460] Lustre: lustre-OST0000: new disk, initializing [ 4968.424029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4968.455056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.359109] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4973.054923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4973.058323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4973.091063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4973.259434] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.146066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4978.146595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4978.148077] LustreError: Skipped 1 previous similar message [ 4978.148552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4978.152982] Lustre: Skipped 1 previous similar message [ 4979.420697] Lustre: server umount lustre-OST0000 complete [ 4982.830213] Lustre: server umount lustre-MDT0000 complete [ 4983.265846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.274086] Lustre: Skipped 1 previous similar message [ 4984.194326] LustreError: 310676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914904 with bad export cookie 11428521454176161777 [ 4984.197551] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.199757] LustreError: 310676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4984.320619] Lustre: server umount lustre-MDT0001 complete [ 4988.098588] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 4989.036742] Key type lgssc unregistered [ 4989.198652] LNet: 314053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4989.203637] LNetError: 314053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4989.217542] LNet: Removed LNI 192.168.202.122@tcp [ 4989.523653] Key type .llcrypt unregistered [ 4989.525051] Key type ._llcrypt unregistered [ 4999.934014] Key type ._llcrypt registered [ 4999.935231] Key type .llcrypt registered [ 4999.969239] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5000.247264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5000.255947] alg: No test for adler32 (adler32-zlib) [ 5001.106693] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 5001.189885] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5002.767192] Key type lgssc registered [ 5003.086613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5006.051112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5006.057256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.227188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.890761] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5013.040785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.177936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5014.550469] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5016.922550] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5018.488714] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5020.902786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.033102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.648749] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5024.227024] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5037.084860] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5037.542527] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5039.071689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5039.075058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.080485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5039.723248] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 5040.848748] Lustre: server umount lustre-MDT0000 complete [ 5041.975326] LustreError: 316150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777914962 with bad export cookie 9201187264304871808 [ 5041.978473] LustreError: 316150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5041.979612] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5042.656111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.656633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5042.661177] Lustre: Skipped 1 previous similar message [ 5042.661685] LustreError: 315427:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.669561] LustreError: 315427:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5044.838352] LustreError: 315429:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.843519] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 5046.687791] LustreError: 316590:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5048.165189] Lustre: server umount lustre-MDT0001 complete [ 5049.963565] Lustre: lustre-OST0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 5049.965780] Lustre: Skipped 1 previous similar message [ 5055.443851] Lustre: server umount lustre-OST0000 complete [ 5057.957881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.067362] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5059.910041] Lustre: 319174:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.122@tcp [ 5060.533189] Lustre: server umount MGS complete [ 5062.962035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.107149] LustreError: 319747:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.126378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.221772] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5067.256624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.453899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.326126] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5073.029843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.606452] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5078.033580] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5080.950472] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 5082.693722] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5083.580161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5087.719664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.721180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.724734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5087.731543] Lustre: Skipped 1 previous similar message [ 5087.734245] Lustre: Skipped 1 previous similar message [ 5091.289791] Lustre: server umount lustre-OST0000 complete [ 5093.344408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5093.349166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.956365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.276142] Lustre: server umount lustre-MDT0000 complete [ 5101.201099] LustreError: 321168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777915021 with bad export cookie 9201187264304872641 [ 5101.208149] LustreError: 321168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5101.209160] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5101.364633] Lustre: server umount lustre-MDT0001 complete [ 5105.984986] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5107.465296] Key type lgssc unregistered [ 5107.643759] LNet: 322918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5107.648508] LNetError: 322918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5107.658798] LNet: Removed LNI 192.168.202.122@tcp [ 5108.080556] Key type .llcrypt unregistered [ 5108.081582] Key type ._llcrypt unregistered [ 5116.291050] Key type ._llcrypt registered [ 5116.292230] Key type .llcrypt registered [ 5116.338080] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5124.023457] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5124.501603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5124.573979] alg: No test for adler32 (adler32-zlib) [ 5125.471962] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 5125.595705] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5127.207160] Key type lgssc registered [ 5127.843347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5144.577494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5144.585292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.714570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5145.727281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5145.770051] Lustre: lustre-MDT0000: new disk, initializing [ 5145.803228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.811604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5147.559622] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5152.323673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.388109] Lustre: 326017:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5152.426958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5152.430453] Lustre: Skipped 1 previous similar message [ 5152.478438] Lustre: lustre-MDT0001: new disk, initializing [ 5152.524445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5152.546587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5152.552182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5154.129563] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5157.345912] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5158.929079] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5162.261350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5162.426640] Lustre: lustre-OST0000: new disk, initializing [ 5162.429883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5162.481603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.987809] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5168.649375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5168.654372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5168.702554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5168.992423] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5170.932164] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5171.060212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5172.660267] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5172.733251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5173.729078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5173.731212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.731954] LustreError: Skipped 1 previous similar message [ 5173.732432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5173.738445] Lustre: Skipped 1 previous similar message [ 5175.515780] Lustre: server umount lustre-OST0000 complete [ 5178.335826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5178.339053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.344473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5178.346065] Lustre: Skipped 1 previous similar message [ 5183.128155] Lustre: server umount lustre-MDT0000 complete [ 5184.481100] LustreError: 326010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777915104 with bad export cookie 3640394519923536768 [ 5184.484170] LustreError: 326010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5184.484679] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5184.597015] Lustre: server umount lustre-MDT0001 complete [ 5188.080643] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:18:27 (1777915107) [ 5194.811544] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5197.602224] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5208.070788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.161135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5208.172125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5208.207262] Lustre: lustre-MDT0000: new disk, initializing [ 5208.228820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.235673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5209.648779] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5214.980358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.055409] Lustre: 332326:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5215.101136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5215.104931] Lustre: Skipped 1 previous similar message [ 5215.160903] Lustre: lustre-MDT0001: new disk, initializing [ 5215.223486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5215.247530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5215.254508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5217.120399] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5221.276135] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5222.875670] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5225.730860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5225.829332] Lustre: lustre-OST0000: new disk, initializing [ 5225.831622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5225.860648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.623641] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5230.670304] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5231.603731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5231.608960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5231.635502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5246.945682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5246.946335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.947062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5246.947067] Lustre: Skipped 1 previous similar message [ 5246.949390] LustreError: Skipped 1 previous similar message [ 5246.953755] Lustre: Skipped 2 previous similar messages [ 5250.635445] Lustre: server umount lustre-OST0000 complete [ 5252.065924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.066542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5252.071273] Lustre: Skipped 1 previous similar message [ 5252.074235] Lustre: Skipped 1 previous similar message [ 5257.184289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.186175] Lustre: Skipped 2 previous similar messages [ 5266.399340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5266.504336] Lustre: server umount lustre-MDT0000 complete [ 5267.425254] LustreError: 333122:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.434633] LustreError: 333122:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5268.136722] LustreError: 332319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777915188 with bad export cookie 3640394519923537650 [ 5268.141603] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5268.290598] Lustre: server umount lustre-MDT0001 complete [ 5272.053111] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5273.477441] Key type lgssc unregistered [ 5273.673775] LNet: 336087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5273.678849] LNetError: 336087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5273.693662] LNet: Removed LNI 192.168.202.122@tcp [ 5274.146494] Key type .llcrypt unregistered [ 5274.148869] Key type ._llcrypt unregistered [ 5313.498648] Key type ._llcrypt registered [ 5313.499852] Key type .llcrypt registered [ 5313.536993] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5313.645433] Key type .llcrypt unregistered [ 5313.646849] Key type ._llcrypt unregistered [ 5349.685505] Key type ._llcrypt registered [ 5349.687152] Key type .llcrypt registered [ 5349.732255] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5349.851169] Key type .llcrypt unregistered [ 5349.852375] Key type ._llcrypt unregistered [ 5360.001508] Key type ._llcrypt registered [ 5360.002541] Key type .llcrypt registered [ 5360.035705] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5360.123203] Key type .llcrypt unregistered [ 5360.124610] Key type ._llcrypt unregistered [ 5370.909402] Key type ._llcrypt registered [ 5370.910835] Key type .llcrypt registered [ 5370.949946] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5371.054322] Key type .llcrypt unregistered [ 5371.055455] Key type ._llcrypt unregistered [ 5397.969370] Key type ._llcrypt registered [ 5397.970650] Key type .llcrypt registered [ 5398.011981] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5398.113348] Key type .llcrypt unregistered [ 5398.114401] Key type ._llcrypt unregistered [ 5408.934579] Key type ._llcrypt registered [ 5408.935625] Key type .llcrypt registered [ 5408.974692] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5409.080242] Key type .llcrypt unregistered [ 5409.081276] Key type ._llcrypt unregistered [ 5419.705576] Key type ._llcrypt registered [ 5419.706685] Key type .llcrypt registered [ 5419.748468] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5419.855220] Key type .llcrypt unregistered [ 5419.856205] Key type ._llcrypt unregistered [ 5430.642411] Key type ._llcrypt registered [ 5430.643397] Key type .llcrypt registered [ 5430.677105] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5430.764337] Key type .llcrypt unregistered [ 5430.765339] Key type ._llcrypt unregistered [ 5442.181859] Key type ._llcrypt registered [ 5442.182847] Key type .llcrypt registered [ 5442.218196] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5442.317300] Key type .llcrypt unregistered [ 5442.318252] Key type ._llcrypt unregistered [ 5452.959593] Key type ._llcrypt registered [ 5452.960526] Key type .llcrypt registered [ 5453.001934] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5453.094265] Key type .llcrypt unregistered [ 5453.095205] Key type ._llcrypt unregistered [ 5473.785353] Key type ._llcrypt registered [ 5473.786324] Key type .llcrypt registered [ 5473.826133] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5473.931169] Key type .llcrypt unregistered [ 5473.932144] Key type ._llcrypt unregistered [ 5483.893214] Key type ._llcrypt registered [ 5483.894138] Key type .llcrypt registered [ 5483.927320] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5484.010289] Key type .llcrypt unregistered [ 5484.011256] Key type ._llcrypt unregistered [ 5494.392451] Key type ._llcrypt registered [ 5494.393363] Key type .llcrypt registered [ 5494.424054] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5494.507303] Key type .llcrypt unregistered [ 5494.508374] Key type ._llcrypt unregistered [ 5515.541741] Key type ._llcrypt registered [ 5515.542724] Key type .llcrypt registered [ 5515.576188] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5521.396767] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5521.643598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5521.653509] alg: No test for adler32 (adler32-zlib) [ 5522.490551] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 5522.565131] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5524.135113] Key type lgssc registered [ 5524.430842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5534.439050] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5534.443454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.541158] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5535.550799] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5535.585835] Lustre: tst250fs-MDT0000: new disk, initializing [ 5535.607176] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.612689] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5536.749293] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5540.442775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.477283] Lustre: 355750:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5540.496296] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5540.497903] Lustre: Skipped 1 previous similar message [ 5540.531135] Lustre: tst250fs-MDT0001: new disk, initializing [ 5540.559909] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5540.569505] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5540.574831] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5541.652050] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5543.754324] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5545.670406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5545.763979] Lustre: tst250fs-OST0000: new disk, initializing [ 5545.765976] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5545.794379] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.402931] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5551.229217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5551.276978] Lustre: tst250fs-OST0001: new disk, initializing [ 5551.278440] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5551.303800] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5552.925852] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5553.790230] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5553.794751] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5553.797532] Lustre: Skipped 1 previous similar message [ 5553.838223] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5553.841852] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5555.271528] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5566.431732] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5566.433815] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.437304] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.504723] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.507168] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.510653] Lustre: Skipped 2 previous similar messages [ 5570.709854] Lustre: server umount tst250fs-MDT0000 complete [ 5571.780902] LustreError: 356973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777915491 with bad export cookie 16085006210329627800 [ 5571.784738] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5571.785914] LustreError: 356973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5571.896908] Lustre: server umount tst250fs-MDT0001 complete [ 5583.888890] Lustre: server umount tst250fs-OST0000 complete [ 5595.835495] Lustre: server umount tst250fs-OST0001 complete [ 5604.097157] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5606.834147] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5621.275922] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:25:40 (1777915540) [ 5622.081570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5629.589049] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5634.554919] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5649.987720] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5458 sec ========== 13:26:09 (1777915569) [ 5650.845280] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:26:10 (1777915570) === [ 5652.545055] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:26:11 (1777915571) === [ 5670.601780] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5672.137623] Key type lgssc unregistered [ 5672.297664] LNet: 367206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5672.302804] LNetError: 367206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5672.311503] LNet: Removed LNI 192.168.202.122@tcp [ 5672.766154] Key type .llcrypt unregistered [ 5672.768491] Key type ._llcrypt unregistered