[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500171315 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003104] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008496] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022802] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027378] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.032024] *** VALIDATE proc *** [ 0.033254] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038034] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042406] debug: unmapping init [mem 0xffffffff8a859000-0xffffffff8a860fff] [ 0.044276] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045783] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055645] smp: Bringing up secondary CPUs ... [ 0.056683] x86: Booting SMP configuration: [ 0.057032] .... node #0, CPUs: #1 #2 #3 [ 0.060513] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.145932] node 0 deferred pages initialised in 79ms [ 0.148119] devtmpfs: initialized [ 0.149297] x86/mm: Memory block size: 128MB [ 0.151954] gcov: version magic: 0x41383552 [ 0.153358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158385] pinctrl core: initialized pinctrl subsystem [ 0.160172] [ 0.160760] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ** ** [ 0.167015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170012] ** ** [ 0.172044] ** This means that this kernel is built to expose internal ** [ 0.175017] ** IOMMU data structures, which may compromise security on ** [ 0.177010] ** your system. ** [ 0.179012] ** ** [ 0.181014] ** If you see this message and you are not debugging the ** [ 0.183016] ** kernel, report this immediately to your vendor! ** [ 0.185012] ** ** [ 0.187013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189012] ************************************************************* [ 0.191866] NET: Registered protocol family 16 [ 0.193465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203534] cpuidle: using governor menu [ 0.205109] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206398] PCI: Using configuration type 1 for base access [ 0.208128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220188] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226135] cryptd: max_cpu_qlen set to 1000 [ 0.229226] ACPI: Added _OSI(Module Device) [ 0.231012] ACPI: Added _OSI(Processor Device) [ 0.233016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239269] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247224] ACPI: Interpreter enabled [ 0.248062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250012] ACPI: Using IOAPIC for interrupt routing [ 0.252105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255423] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279364] acpiphp: Slot [2] registered [ 0.280122] acpiphp: Slot [5] registered [ 0.282108] acpiphp: Slot [6] registered [ 0.283097] acpiphp: Slot [7] registered [ 0.284096] acpiphp: Slot [8] registered [ 0.286110] acpiphp: Slot [9] registered [ 0.287094] acpiphp: Slot [10] registered [ 0.288080] acpiphp: Slot [3] registered [ 0.290124] acpiphp: Slot [4] registered [ 0.291117] acpiphp: Slot [11] registered [ 0.292074] acpiphp: Slot [12] registered [ 0.294120] acpiphp: Slot [13] registered [ 0.296128] acpiphp: Slot [14] registered [ 0.297108] acpiphp: Slot [15] registered [ 0.299130] acpiphp: Slot [16] registered [ 0.301102] acpiphp: Slot [17] registered [ 0.302080] acpiphp: Slot [18] registered [ 0.303094] acpiphp: Slot [19] registered [ 0.305092] acpiphp: Slot [20] registered [ 0.306120] acpiphp: Slot [21] registered [ 0.307106] acpiphp: Slot [22] registered [ 0.309103] acpiphp: Slot [23] registered [ 0.310092] acpiphp: Slot [24] registered [ 0.311083] acpiphp: Slot [25] registered [ 0.312100] acpiphp: Slot [26] registered [ 0.314105] acpiphp: Slot [27] registered [ 0.315090] acpiphp: Slot [28] registered [ 0.316114] acpiphp: Slot [29] registered [ 0.318093] acpiphp: Slot [30] registered [ 0.319117] acpiphp: Slot [31] registered [ 0.321060] PCI host bridge to bus 0000:00 [ 0.322025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338977] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342261] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.357491] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.364018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.366568] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370906] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377749] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403840] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408712] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458898] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503536] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538036] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549918] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.554033] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590393] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.598021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.607022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628031] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641483] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.675026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.690360] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.697282] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.703675] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706472] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709336] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.716035] iommu: Default domain type: Passthrough [ 0.718455] SCSI subsystem initialized [ 0.720195] ACPI: bus type USB registered [ 0.722208] usbcore: registered new interface driver usbfs [ 0.725086] usbcore: registered new interface driver hub [ 0.726076] usbcore: registered new device driver usb [ 0.728220] pps_core: LinuxPPS API ver. 1 registered [ 0.731014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735114] PTP clock support registered [ 0.737233] EDAC MC: Ver: 3.0.0 [ 0.740114] PCI: Using ACPI for IRQ routing [ 0.743090] NetLabel: Initializing [ 0.744017] NetLabel: domain hash size = 128 [ 0.746023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.748148] NetLabel: unlabeled traffic allowed by default [ 0.751343] vgaarb: loaded [ 0.753412] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.762004] clocksource: Switched to clocksource kvm-clock [ 0.877095] VFS: Disk quotas dquot_6.6.0 [ 0.878931] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883171] *** VALIDATE ramfs *** [ 0.884627] *** VALIDATE hugetlbfs *** [ 0.886616] pnp: PnP ACPI init [ 0.889359] pnp: PnP ACPI: found 6 devices [ 0.909805] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.913274] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.915639] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917943] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.920723] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.923313] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.926271] NET: Registered protocol family 2 [ 0.928943] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.935243] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939192] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.944949] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.948629] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951937] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955458] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.958550] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.961815] NET: Registered protocol family 1 [ 0.965087] RPC: Registered named UNIX socket transport module. [ 0.967447] RPC: Registered udp transport module. [ 0.969412] RPC: Registered tcp transport module. [ 0.971323] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.973774] NET: Registered protocol family 44 [ 0.975654] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.978036] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.980383] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.982976] PCI: CLS 0 bytes, default 64 [ 0.985266] Unpacking initramfs... [ 2.417139] debug: unmapping init [mem 0xffff942c7cc54000-0xffff942c7ffbffff] [ 2.421177] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.423755] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.427160] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.945911] Initialise system trusted keyrings [ 2.947800] Key type blacklist registered [ 2.949860] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.960578] zbud: loaded [ 2.963842] *** VALIDATE nfs *** [ 2.965286] *** VALIDATE nfs4 *** [ 2.967919] pstore: using deflate compression [ 2.971566] Platform Keyring initialized [ 3.082312] NET: Registered protocol family 38 [ 3.084084] Key type asymmetric registered [ 3.085827] Asymmetric key parser 'x509' registered [ 3.087547] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.090321] io scheduler mq-deadline registered [ 3.092389] io scheduler kyber registered [ 3.094242] io scheduler bfq registered [ 3.096192] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.099403] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.102463] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.105445] ACPI: Power Button [PWRF] [ 3.110946] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.118294] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.130253] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.138401] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.152156] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.182082] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.211379] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.215638] Non-volatile memory driver v1.3 [ 3.217657] Linux agpgart interface v0.103 [ 3.255914] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.258795] vda: detected capacity change from 0 to 74702848 [ 3.286844] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.290651] vdb: detected capacity change from 0 to 1073741824 [ 3.307277] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310243] vdc: detected capacity change from 0 to 2621440000 [ 3.334402] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.337404] vdd: detected capacity change from 0 to 2621440000 [ 3.358610] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362230] vde: detected capacity change from 0 to 4294967296 [ 3.379582] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.382898] vdf: detected capacity change from 0 to 4294967296 [ 3.390692] libphy: Fixed MDIO Bus: probed [ 3.396384] usbcore: registered new interface driver usbserial_generic [ 3.399231] usbserial: USB Serial support registered for generic [ 3.402227] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.407806] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.410303] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.412618] mousedev: PS/2 mouse device common for all mice [ 3.416345] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.416347] rtc_cmos 00:05: RTC can wake from S4 [ 3.419120] rtc_cmos 00:05: registered as rtc0 [ 3.425629] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.425863] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.427981] intel_pstate: CPU model not supported [ 3.435208] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.438961] hid: raw HID events driver (C) Jiri Kosina [ 3.442261] usbcore: registered new interface driver usbhid [ 3.444587] usbhid: USB HID core driver [ 3.446164] drop_monitor: Initializing network drop monitor service [ 3.448817] Initializing XFRM netlink socket [ 3.451348] NET: Registered protocol family 10 [ 3.454577] Segment Routing with IPv6 [ 3.456532] NET: Registered protocol family 17 [ 3.458965] mpls_gso: MPLS GSO support [ 3.463444] RAS: Correctable Errors collector initialized. [ 3.465260] AVX version of gcm_enc/dec engaged. [ 3.467023] AES CTR mode by8 optimization enabled [ 3.550445] sched_clock: Marking stable (3550370755, 0)->(4475202156, -924831401) [ 3.553316] registered taskstats version 1 [ 3.556205] Loading compiled-in X.509 certificates [ 3.558729] zswap: loaded using pool lzo/zbud [ 3.585054] Key type big_key registered [ 3.598388] Key type encrypted registered [ 3.600056] ima: No TPM chip found, activating TPM-bypass! [ 3.602308] ima: Allocated hash algorithm: sha1 [ 3.604827] ima: No architecture policies found [ 3.606640] evm: Initialising EVM extended attributes: [ 3.608746] evm: security.selinux [ 3.610198] evm: security.ima [ 3.611336] evm: security.capability [ 3.612752] evm: HMAC attrs: 0x1 [ 3.615641] rtc_cmos 00:05: setting system clock to 2026-08-17 13:56:42 UTC (1786975002) [ 3.622678] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.626903] debug: unmapping init [mem 0xffffffff8a582000-0xffffffff8a858fff] [ 3.635163] Write protecting the kernel read-only data: 28672k [ 3.638979] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.641930] debug: unmapping init [mem 0xffffffff89514000-0xffffffff895fffff] [ 3.675266] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.684363] systemd[1]: Detected virtualization kvm. [ 3.686216] systemd[1]: Detected architecture x86-64. [ 3.688078] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.717783] systemd[1]: No hostname configured. [ 3.719544] systemd[1]: Set hostname to . [ 3.721624] random: systemd: uninitialized urandom read (16 bytes read) [ 3.723934] systemd[1]: Initializing machine ID from random generator. [ 3.762695] random: ln: uninitialized urandom read (6 bytes read) [ 3.854634] random: systemd: uninitialized urandom read (16 bytes read) [ 3.857662] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.862932] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.868247] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.541918] device-mapper: uevent: version 1.0.3 [ 4.544443] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.366755] random: fast init done [ 5.373912] virtio_net virtio0 ens2: renamed from eth0 [ 5.431709] scsi host0: ata_piix [ 5.458898] scsi host1: ata_piix [ 5.460672] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.463584] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.123586] random: crng init done [ 10.126230] random: 7 urandom warning(s) missed due to ratelimiting [ 10.538742] dracut-initqueue[602]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.254712] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ 14.161374] hrtimer: interrupt took 4994328 ns [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.425202] printk: systemd: 25 output lines suppressed due to ratelimiting [ 16.368344] SELinux: Disabled at runtime. [ 16.476909] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.488707] systemd[1]: Detected virtualization kvm. [ 16.491089] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.187583] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.193572] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.204487] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.218260] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.228473] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.263038] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.281943] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 18.760382] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 19.830652] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.845009] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.017215] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.407580] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.447641] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 25.961449] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.787519] NFS: Registering the id_resolver key type [ 26.790269] Key type id_resolver registered [ 26.794617] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg419-server login: [ 89.233634] libcfs: loading out-of-tree module taints kernel. [ 89.387278] Key type ._llcrypt registered [ 89.395499] Key type .llcrypt registered [ 89.567300] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 110.164809] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 113.092951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.121362] alg: No test for adler32 (adler32-zlib) [ 114.760505] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 115.866808] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 117.592226] Key type lgssc registered [ 119.523891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.567191] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.164700] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 192.689555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.714905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.097611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.131609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.213565] Lustre: lustre-MDT0000: new disk, initializing [ 194.336518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.354783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.304098] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.819898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.996081] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.055360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 214.062176] Lustre: Skipped 1 previous similar message [ 214.176559] Lustre: lustre-MDT0001: new disk, initializing [ 214.256456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.285629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.298540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.176840] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.018190] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.834702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.063547] Lustre: lustre-OST0000: new disk, initializing [ 234.073164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.082436] Lustre: 8463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.161250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.184351] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.284204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.306467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.400455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.729528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.897237] Lustre: lustre-OST0001: new disk, initializing [ 257.913632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 257.926353] Lustre: 9539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.087195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.825237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.852878] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.913914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 265.939637] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.526116] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.722656] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.125266] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.854627] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing yml_node [ 304.798496] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 307.645843] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 310.845449] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 312.650461] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 10:01:49 EDT 2026 [ 332.161947] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 333.678510] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.225757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.234435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.248748] Lustre: Skipped 1 previous similar message [ 352.266396] Lustre: Skipped 3 previous similar messages [ 356.277257] Lustre: server umount lustre-MDT0000 complete [ 360.359269] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975359 with bad export cookie 13008562696795683070 [ 360.360338] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.375832] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 362.463869] LustreError: 8471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.470296] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.489506] LustreError: 8471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 362.530671] Lustre: Skipped 2 previous similar messages [ 362.535550] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.549897] Lustre: Skipped 1 previous similar message [ 365.537766] LustreError: 6534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.537842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.562212] LustreError: 6534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 365.575884] Lustre: Skipped 1 previous similar message [ 366.857942] Lustre: server umount lustre-MDT0001 complete [ 371.628771] Lustre: server umount lustre-OST0000 complete [ 375.731050] Lustre: server umount lustre-OST0001 complete [ 383.396740] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 393.668299] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 430.869684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.120254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 431.162881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 431.266485] Lustre: lustre-MDT0000: new disk, initializing [ 431.344715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.374567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.637378] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 448.155110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.377548] Lustre: 15104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 448.469486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 448.473590] Lustre: Skipped 1 previous similar message [ 448.621477] Lustre: lustre-MDT0001: new disk, initializing [ 448.781801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 448.829118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 448.853553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.773094] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 463.775934] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 469.913276] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.460828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.821679] Lustre: lustre-OST0000: new disk, initializing [ 477.831469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.840418] Lustre: 17076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.961916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.904234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.913530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 480.084274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 484.752873] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 494.028811] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.693649] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.866806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.980696] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.195848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.827981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.831297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.841940] Lustre: Skipped 1 previous similar message [ 505.854489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.952198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.957247] Lustre: Skipped 2 previous similar messages [ 519.135167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.252814] Lustre: server umount lustre-OST0000 complete [ 525.794310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.801506] LustreError: Skipped 1 previous similar message [ 525.808969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.828330] Lustre: Skipped 1 previous similar message [ 525.844940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 525.857639] Lustre: Skipped 2 previous similar messages [ 528.790490] Lustre: server umount lustre-MDT0000 complete [ 531.427444] LustreError: 15116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.442597] LustreError: 15116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 531.914127] LustreError: 15903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975530 with bad export cookie 13008562696795684603 [ 531.928915] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 531.931927] LustreError: 15903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 532.389404] Lustre: server umount lustre-MDT0001 complete [ 538.761955] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:05:35 (1786975535) [ 558.345140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.841328] LustreError: 19953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.898113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.747453] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 564.195938] LustreError: 19954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.136720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.079917] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 584.853856] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 590.098231] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 592.163700] Lustre: Failing over lustre-MDT0000 [ 592.172166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.181732] Lustre: Skipped 1 previous similar message [ 592.186765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.189703] Lustre: Skipped 3 previous similar messages [ 592.421302] Lustre: server umount lustre-MDT0000 complete [ 597.481129] LustreError: 20958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.500356] LustreError: 20958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 611.932984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.040721] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.415380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 616.679180] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 617.443970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.452166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 617.475589] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 626.923812] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 632.810490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.815337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.819034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.846180] Lustre: Skipped 2 previous similar messages [ 637.256499] Lustre: server umount lustre-MDT0000 complete [ 637.921357] LustreError: 22529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.946914] LustreError: 22529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 641.559337] LustreError: 19934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975640 with bad export cookie 13008562696795686017 [ 641.573364] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.585747] LustreError: 19934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 641.952955] Lustre: server umount lustre-MDT0001 complete [ 650.443670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.088701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.095037] Lustre: Skipped 2 previous similar messages [ 655.718976] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 663.673087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.908084] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 674.222996] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 679.020416] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 681.018447] Lustre: Failing over lustre-MDT0000 [ 681.279534] Lustre: server umount lustre-MDT0000 complete [ 684.513888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 684.526072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.539209] LustreError: 23293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.548375] LustreError: 23293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 698.998907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.105929] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.352572] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 703.553285] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 704.485288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 704.506603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 704.518099] Lustre: Skipped 1 previous similar message [ 704.541929] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 712.354415] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 719.841580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 719.844573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.866628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.881711] Lustre: Skipped 2 previous similar messages [ 719.908220] Lustre: Skipped 2 previous similar messages [ 722.172626] Lustre: server umount lustre-MDT0000 complete [ 725.116746] LustreError: 23274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975724 with bad export cookie 13008562696795686955 [ 725.123346] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.128520] LustreError: 23274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 725.407600] Lustre: server umount lustre-MDT0001 complete [ 733.460472] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:08:50 (1786975730) [ 749.704527] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 761.534598] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 804.564707] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 815.308910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.639626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 815.679800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 815.791091] Lustre: lustre-MDT0000: new disk, initializing [ 815.912302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.916801] Lustre: Skipped 2 previous similar messages [ 815.952604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 820.967414] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 830.197711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.317174] Lustre: 29669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 830.610068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 834.603490] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 842.745657] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 847.605176] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 853.266150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.491037] Lustre: 31638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 855.192234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 855.267850] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 855.313929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 855.337048] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 859.409405] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 867.873830] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 873.170363] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 896.485135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 896.497910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.525720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 896.531422] Lustre: Skipped 1 previous similar message [ 898.599775] Lustre: server umount lustre-OST0000 complete [ 907.999263] Lustre: server umount lustre-MDT0000 complete [ 911.693519] LustreError: 29661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975910 with bad export cookie 13008562696795687361 [ 911.705060] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.709858] LustreError: 29661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 912.153956] Lustre: server umount lustre-MDT0001 complete [ 922.429027] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 925.882957] Key type lgssc unregistered [ 926.267461] LNet: 33616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.283513] LNetError: 33616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.316609] LNet: Removed LNI 192.168.204.119@tcp [ 927.494466] Key type .llcrypt unregistered [ 927.496242] Key type ._llcrypt unregistered [ 951.818646] Key type ._llcrypt registered [ 951.825559] Key type .llcrypt registered [ 951.977592] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 953.129670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 953.262293] alg: No test for adler32 (adler32-zlib) [ 954.418427] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 954.763693] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 956.439238] Key type lgssc registered [ 957.711660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 964.731957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 964.755371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.179348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.207404] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 976.358228] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.376886] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 977.858623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.084035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.003851] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 988.879797] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 996.409199] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1004.742745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.555857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.645568] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1019.876184] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1022.961968] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1035.132630] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:13:52 (1786976032) [ 1036.903566] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1038.589822] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1045.372699] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:14:02 (1786976042) [ 1053.515917] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:14:10 (1786976050) [ 1054.741934] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1063.738022] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1063.754217] Lustre: Skipped 2 previous similar messages [ 1065.751397] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:14:22 (1786976062) [ 1069.461973] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1069.469843] Lustre: Skipped 3 previous similar messages [ 1086.468525] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1086.474964] Lustre: Skipped 6 previous similar messages [ 1089.068840] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:14:45 (1786976085) [ 1092.367465] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1096.021676] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1104.201144] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:15:01 (1786976101) [ 1105.803463] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1105.808426] Lustre: Skipped 3 previous similar messages [ 1118.300199] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:15:15 (1786976115) [ 1134.560500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1134.571211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.596116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.590461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1135.594146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.596569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.614343] Lustre: Skipped 1 previous similar message [ 1136.712129] Lustre: server umount lustre-OST0000 complete [ 1142.240569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.250867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.293144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1145.832589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.832859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1145.859670] Lustre: Skipped 1 previous similar message [ 1147.221907] Lustre: server umount lustre-MDT0000 complete [ 1150.945980] LustreError: 35027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.959357] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976149 with bad export cookie 17993584808569605636 [ 1150.968287] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.968369] LustreError: 35027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1151.012321] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.393715] Lustre: server umount lustre-MDT0001 complete [ 1165.124660] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1169.020337] Key type lgssc unregistered [ 1169.457264] LNet: 42893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.463078] LNetError: 42893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.541399] LNet: Removed LNI 192.168.204.119@tcp [ 1171.669465] Key type .llcrypt unregistered [ 1171.672993] Key type ._llcrypt unregistered [ 1193.707317] Key type ._llcrypt registered [ 1193.714414] Key type .llcrypt registered [ 1193.838462] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 1212.975050] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1214.639291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.699830] alg: No test for adler32 (adler32-zlib) [ 1216.027825] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1216.241837] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1217.863147] Key type lgssc registered [ 1219.054032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.309352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.331406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.489030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1261.556216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1261.702591] Lustre: lustre-MDT0000: new disk, initializing [ 1261.869703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.909229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1267.011392] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1279.697474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.783846] Lustre: 46052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1279.830520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1279.833751] Lustre: Skipped 1 previous similar message [ 1279.897883] Lustre: lustre-MDT0001: new disk, initializing [ 1280.027734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.078524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1280.104320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1284.873626] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1295.733720] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1301.873673] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1310.709301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.105278] Lustre: lustre-OST0000: new disk, initializing [ 1311.111418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1311.118278] Lustre: 48021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1311.234691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.524644] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1319.616908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1319.637439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1319.836402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1328.836984] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1333.473384] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1333.754734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1338.824668] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1339.106167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1345.511173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1345.514180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.525553] LustreError: Skipped 1 previous similar message [ 1345.528157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1345.548088] Lustre: Skipped 1 previous similar message [ 1347.682318] Lustre: server umount lustre-OST0000 complete [ 1351.648141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1351.655447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.689587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.693154] Lustre: Skipped 1 previous similar message [ 1355.745692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.745805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.943299] Lustre: server umount lustre-MDT0000 complete [ 1361.768681] LustreError: 46043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976360 with bad export cookie 10592711271425074553 [ 1361.772579] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.779368] LustreError: 46043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1362.138337] Lustre: server umount lustre-MDT0001 complete [ 1371.759126] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:19:28 (1786976368) [ 1379.102162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.591470] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.619157] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1379.691715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.004076] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1384.928062] LustreError: 50485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.054192] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.529160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.069496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.439841] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1406.140893] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.527661] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1420.621145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.974862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.909210] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1434.759781] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.834281] LustreError: 53013:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1444.181394] Lustre: Modifying parameter general.timeout=1 in log params [ 1445.228721] Lustre: Modifying parameter general.timeout=2 in log params [ 1447.589843] Lustre: Modifying parameter general.timeout=5 in log params [ 1447.592896] Lustre: Skipped 2 previous similar messages [ 1452.323970] Lustre: Modifying parameter general.timeout=10 in log params [ 1452.334271] Lustre: Skipped 4 previous similar messages [ 1461.264472] Lustre: Modifying parameter general.timeout=19 in log params [ 1461.267247] Lustre: Skipped 8 previous similar messages [ 1477.910284] Lustre: Modifying parameter general.timeout=37 in log params [ 1477.922346] Lustre: Skipped 17 previous similar messages [ 1510.196207] Lustre: Modifying parameter general.timeout=72 in log params [ 1510.200711] Lustre: Skipped 34 previous similar messages [ 1574.544401] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:22:51 (1786976571) [ 1575.551512] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1575.557731] Lustre: Skipped 58 previous similar messages [ 1577.467829] Lustre: 59957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1579.197836] Lustre: 60064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1582.899159] Lustre: 60266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1582.910349] Lustre: 60266:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1616.821848] Lustre: 61838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1616.830523] Lustre: 61838:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1616.858526] LustreError: 61838:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1621.243744] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:23:37 (1786976617) [ 1638.881973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.894590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.911048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.924434] Lustre: Skipped 1 previous similar message [ 1642.675195] Lustre: server umount lustre-MDT0000 complete [ 1646.054498] LustreError: 51212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.071948] LustreError: 51212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1646.686089] LustreError: 51219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976645 with bad export cookie 10592711271425075435 [ 1646.691160] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.706213] LustreError: 51219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1647.136590] Lustre: server umount lustre-MDT0001 complete [ 1661.834041] Lustre: server umount lustre-OST0000 complete [ 1693.487026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.960541] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1694.449379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.703666] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1709.222756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.451378] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1709.525249] Lustre: 64372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1714.671244] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1722.543668] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1728.889923] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1736.188871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.393307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1736.694397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.705049] Lustre: Skipped 1 previous similar message [ 1744.218624] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1752.244985] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1754.012380] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1754.019125] Lustre: Skipped 22 previous similar messages [ 1754.051271] Lustre: 66940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1764.322363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.323899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.331897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1764.331905] Lustre: Skipped 3 previous similar messages [ 1764.356321] Lustre: Skipped 3 previous similar messages [ 1768.076248] Lustre: server umount lustre-OST0000 complete [ 1774.560650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.564358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1774.574489] Lustre: Skipped 1 previous similar message [ 1774.596175] Lustre: Skipped 2 previous similar messages [ 1778.508985] Lustre: server umount lustre-MDT0000 complete [ 1779.690048] LustreError: 64382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.726423] LustreError: 64382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1782.839872] LustreError: 65114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976781 with bad export cookie 10592711271425078529 [ 1782.847360] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1782.852027] LustreError: 65114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1783.344763] Lustre: server umount lustre-MDT0001 complete [ 1796.345858] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1799.773723] Key type lgssc unregistered [ 1800.212773] LNet: 68150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.223715] LNetError: 68150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.245257] LNet: Removed LNI 192.168.204.119@tcp [ 1801.353390] Key type .llcrypt unregistered [ 1801.356272] Key type ._llcrypt unregistered [ 1832.766910] Key type ._llcrypt registered [ 1832.769534] Key type .llcrypt registered [ 1832.908183] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1834.398101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.654104] alg: No test for adler32 (adler32-zlib) [ 1835.818339] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1836.213576] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1837.905729] Key type lgssc registered [ 1839.488829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.938145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.970330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.723257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.348416] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1861.095923] LustreError: 69569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.116740] LustreError: 69569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1866.213684] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.239939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.823162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1872.062917] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1880.047347] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.647432] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1893.775709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.383927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.068353] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1908.625213] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.900484] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1927.143281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.159720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.161316] Lustre: Skipped 1 previous similar message [ 1927.179169] Lustre: Skipped 2 previous similar messages [ 1932.258844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.272600] Lustre: Skipped 2 previous similar messages [ 1933.862510] Lustre: server umount lustre-MDT0000 complete [ 1937.380993] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.413286] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1938.919333] LustreError: 69551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976937 with bad export cookie 7304048623625927816 [ 1938.923436] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.937670] LustreError: 69551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1939.609285] Lustre: server umount lustre-MDT0001 complete [ 1954.875830] Lustre: server umount lustre-OST0000 complete [ 1984.490693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.828879] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1984.886165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1985.210646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.608398] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2000.352764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.511660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2000.577080] Lustre: 74444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2001.253899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.214976] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2015.216415] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.716748] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2029.890852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2030.095173] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2030.497792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.315230] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2046.018782] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2047.778262] Lustre: 77019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2049.702045] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2058.720038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.727442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.750581] Lustre: Skipped 1 previous similar message [ 2058.754385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.240729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2061.238350] Lustre: server umount lustre-OST0000 complete [ 2067.429435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2067.436224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.447289] Lustre: Skipped 1 previous similar message [ 2067.455356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.460140] Lustre: Skipped 2 previous similar messages [ 2069.476378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.578372] Lustre: server umount lustre-MDT0000 complete [ 2074.595730] LustreError: 77431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.613928] LustreError: 77431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2075.099665] LustreError: 74436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977073 with bad export cookie 7304048623625928656 [ 2075.113464] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2075.117383] LustreError: 74436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2075.492291] Lustre: server umount lustre-MDT0001 complete [ 2087.556968] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2090.652585] Key type lgssc unregistered [ 2091.154891] LNet: 78234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.167376] LNetError: 78234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.205079] LNet: Removed LNI 192.168.204.119@tcp [ 2092.345501] Key type .llcrypt unregistered [ 2092.351449] Key type ._llcrypt unregistered [ 2122.393710] Key type ._llcrypt registered [ 2122.397856] Key type .llcrypt registered [ 2122.601435] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2123.970208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.000270] alg: No test for adler32 (adler32-zlib) [ 2125.137247] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2125.366280] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2127.041314] Key type lgssc registered [ 2128.240075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2136.596079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2136.622207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.103142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.812573] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2147.299610] LustreError: 79657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.314964] LustreError: 79657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2151.988614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.798053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.862104] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2166.936671] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.781588] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.499150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.996663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.645828] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2192.003422] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.517383] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2202.189347] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2203.634422] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2203.637909] Lustre: Skipped 1 previous similar message [ 2205.851717] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2205.855735] Lustre: Skipped 2 previous similar messages [ 2210.253343] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2210.263422] Lustre: Skipped 4 previous similar messages [ 2218.290785] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2218.295916] Lustre: Skipped 7 previous similar messages [ 2234.453382] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2234.458665] Lustre: Skipped 15 previous similar messages [ 2267.027473] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2267.037846] Lustre: Skipped 31 previous similar messages [ 2309.660137] Lustre: 87770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2312.597435] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:35:09 (1786977309) [ 2314.764272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2317.129379] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:35:13 (1786977313) [ 2325.472558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.481369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.490319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.506354] Lustre: Skipped 1 previous similar message [ 2325.530891] Lustre: Skipped 1 previous similar message [ 2330.596732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.608256] Lustre: Skipped 1 previous similar message [ 2335.729066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.746698] Lustre: Skipped 1 previous similar message [ 2336.223134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2336.473595] Lustre: server umount lustre-OST0000 complete [ 2340.833295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.852885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.857579] Lustre: Skipped 1 previous similar message [ 2345.954409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2345.969350] Lustre: Skipped 1 previous similar message [ 2347.141089] Lustre: server umount lustre-MDT0000 complete [ 2351.073910] LustreError: 80403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.095667] LustreError: 80403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2351.389200] LustreError: 79640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977350 with bad export cookie 8845621873020721148 [ 2351.393700] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2351.402368] LustreError: 79640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2351.937535] Lustre: server umount lustre-MDT0001 complete [ 2366.443897] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2370.410345] Key type lgssc unregistered [ 2370.785824] LNet: 89024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2370.800805] LNetError: 89024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2370.822282] LNet: Removed LNI 192.168.204.119@tcp [ 2371.990542] Key type .llcrypt unregistered [ 2371.993536] Key type ._llcrypt unregistered [ 2378.314179] Key type ._llcrypt registered [ 2378.319492] Key type .llcrypt registered [ 2378.576928] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2385.813542] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules [ 2388.378614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.440652] alg: No test for adler32 (adler32-zlib) [ 2389.863785] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2390.181463] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2391.888893] Key type lgssc registered [ 2393.601718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2395.318606] LustreError: 90277:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2415.514565] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2417.376719] LustreError: 90277:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2426.589698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.604169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.194163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.966355] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2437.603399] LustreError: 91298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.635710] LustreError: 91298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2442.747219] LustreError: 91299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.970209] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:37:19 (1786977439) [ 2447.846495] LustreError: 91298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.098709] LustreError: 91330:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2451.111976] LustreError: 91330:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2451.119921] LustreError: 91330:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2451.986109] Lustre: server umount lustre-MDT0000 complete [ 2471.224818] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2475.934478] Key type lgssc unregistered [ 2476.324132] LNet: 92830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2476.334171] LNetError: 92830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.351945] LNet: Removed LNI 192.168.204.119@tcp [ 2477.474495] Key type .llcrypt unregistered [ 2477.482511] Key type ._llcrypt unregistered [ 2497.607429] Key type ._llcrypt registered [ 2497.612231] Key type .llcrypt registered [ 2497.705940] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2498.844752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2498.908987] alg: No test for adler32 (adler32-zlib) [ 2500.043937] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2500.377262] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2502.120662] Key type lgssc registered [ 2503.414753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2512.181175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2512.211059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.720436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.352556] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2524.133757] LustreError: 94049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.162170] LustreError: 94049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2527.111039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.866561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2532.523842] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2540.806911] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2547.348722] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2555.023497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.718482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.470174] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2571.134853] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2626.515325] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:40:23 (1786977623) [ 2628.871490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2631.519479] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:40:27 (1786977627) [ 2637.795355] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.806928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.820040] Lustre: Skipped 1 previous similar message [ 2637.854110] Lustre: Skipped 2 previous similar messages [ 2642.814567] Lustre: server umount lustre-MDT0000 complete [ 2642.919299] LustreError: 95070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.946133] LustreError: 95070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2646.810117] LustreError: 94029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977645 with bad export cookie 11070382058445956207 [ 2646.815459] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.819721] LustreError: 94029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2647.216106] Lustre: server umount lustre-MDT0001 complete [ 2661.897995] Lustre: server umount lustre-OST0000 complete [ 2675.151086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.918016] LustreError: 98350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.164602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.316703] LustreError: 98351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.333071] LustreError: 98351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2681.388740] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2685.422239] LustreError: 98350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.828134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.478746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.196945] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2705.754871] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2712.670806] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2727.539418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.849834] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2727.858253] LustreError: 98340:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2727.893539] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2727.908647] LustreError: 100436:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2727.913985] LustreError: 100436:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2727.922511] LustreError: 100436:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2728.013990] Lustre: server umount lustre-OST0000 complete [ 2728.016955] LustreError: 100436:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2737.521769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.784552] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2737.799817] LustreError: 98338:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2737.816487] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2737.825334] LustreError: 100897:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2737.833418] LustreError: 100897:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2737.838207] LustreError: 100897:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2737.925227] Lustre: server umount lustre-OST0000 complete [ 2737.933246] LustreError: 100897:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2750.449515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.682292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2750.699149] Lustre: Found index 0 for lustre-OST0000, updating log [ 2750.713377] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2750.845520] Lustre: lustre-OST0000: new disk, initializing [ 2750.859141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2750.872643] Lustre: 101431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2751.497776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.018614] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2765.192640] Lustre: Failing over lustre-OST0000 [ 2765.535558] Lustre: server umount lustre-OST0000 complete [ 2772.961451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2772.975739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.001079] Lustre: Skipped 1 previous similar message [ 2773.016820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.520914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.520922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.551895] Lustre: Skipped 1 previous similar message [ 2776.944072] Lustre: server umount lustre-MDT0000 complete [ 2780.650596] LustreError: 98350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.682712] LustreError: 98350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2781.602358] LustreError: 98330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977780 with bad export cookie 11070382058445958951 [ 2781.607768] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2781.610413] LustreError: 98330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2792.197883] Lustre: server umount lustre-MDT0001 complete [ 2804.291829] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:43:20 (1786977800) [ 2827.211644] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2841.711033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.401783] LustreError: 103794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.603509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.113356] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2858.581820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.123833] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2870.060755] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2878.523139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.173522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.185086] Lustre: Skipped 1 previous similar message [ 2879.211202] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2883.238659] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2883.296502] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2883.300192] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2887.383457] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2898.168322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.280336] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2898.352874] Lustre: lustre-OST0001: new disk, initializing [ 2898.355898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2898.362829] Lustre: 106113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2906.786855] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2908.377891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2908.399644] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2908.565757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2916.235309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2925.204885] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2930.622502] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 2931.680293] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2931.692319] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.701152] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 2935.712475] Lustre: Skipped 4 previous similar messages [ 2936.484179] Lustre: server umount lustre-MDT0001 complete [ 2939.385758] LustreError: 103794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.416794] LustreError: 103794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2948.246203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.400717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2948.407674] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2948.536624] Lustre: 103783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2949.474727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.495327] Lustre: Skipped 1 previous similar message [ 2950.392246] LustreError: 107663:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2950.399761] LustreError: 107663:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2950.409961] LustreError: 103786:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2950.561780] LustreError: 107671:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.119@tcp: cfg command failed: rc = -17 [ 2950.576290] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2950.576290] [ 2954.741149] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2955.969482] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2969.473487] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:46:05 (1786977965) [ 2992.144077] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3013.698432] Lustre: 109732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3039.590193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3043.539495] Lustre: 110867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3133.410553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3133.419321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.425979] Lustre: Skipped 2 previous similar messages [ 3133.428818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.678114] Lustre: server umount lustre-MDT0000 complete [ 3139.049612] LustreError: 103791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.067488] LustreError: 103791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3141.982243] LustreError: 107606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978140 with bad export cookie 11070382058445959882 [ 3141.982940] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.988327] LustreError: 107606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3142.343546] Lustre: server umount lustre-MDT0001 complete [ 3157.546120] Lustre: server umount lustre-OST0000 complete [ 3160.415703] Lustre: 93360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786978143/real 1786978143] req@ffff942bc744e300 x1873781718363008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786978159 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3160.443958] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.464831] Lustre: Skipped 3 previous similar messages [ 3161.552687] Lustre: server umount lustre-OST0001 complete [ 3168.769112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3179.345142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3187.692841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3196.979833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3217.153874] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3228.823460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.842249] Lustre: lustre-MDT0000: reset Object Index mappings [ 3229.426088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.728506] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3245.404543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.441105] Lustre: lustre-MDT0001: reset Object Index mappings [ 3252.188785] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3255.258668] Lustre: 115965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3262.147940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3270.252104] LustreError: 116321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.265395] LustreError: 116321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3270.281053] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3271.155534] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3275.374016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3280.165358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.025715] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3286.028037] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3286.425834] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3294.685092] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3298.807675] Lustre: 117842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3351.544039] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:52:28 (1786978348) [ 3379.169585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3379.186551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.211151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3379.233593] Lustre: Skipped 3 previous similar messages [ 3385.346037] Lustre: server umount lustre-MDT0000 complete [ 3389.848716] LustreError: 115968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978388 with bad export cookie 11070382058446006775 [ 3389.859261] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.863964] LustreError: 115968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3404.255219] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3404.730748] Lustre: server umount lustre-MDT0001 complete [ 3415.904461] Lustre: server umount lustre-OST0000 complete [ 3426.415585] Lustre: server umount lustre-OST0001 complete [ 3434.904389] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 3447.371556] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3493.313087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.702192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3493.736281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3493.811903] Lustre: lustre-MDT0000: new disk, initializing [ 3493.934033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.949085] Lustre: Skipped 3 previous similar messages [ 3493.963379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3499.567827] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3507.519512] Lustre: server umount lustre-MDT0000 complete [ 3517.527863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.839455] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3517.857773] Lustre: 122117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3517.870374] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3522.951134] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3528.793825] Lustre: server umount lustre-MDT0000 complete [ 3538.568326] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:55:35 (1786978535) [ 3540.323766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3542.193400] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:55:39 (1786978539) [ 3561.650740] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3572.064844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.910361] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3586.170919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.353882] Lustre: 124170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3586.398642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3586.403267] Lustre: Skipped 1 previous similar message [ 3586.516685] Lustre: lustre-MDT0001: new disk, initializing [ 3586.708057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3586.721601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3591.310278] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3596.409336] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3603.577382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.976449] Lustre: lustre-OST0000: new disk, initializing [ 3603.981694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3603.998460] Lustre: 125747:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3605.523723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3605.542705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3605.690234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3610.756307] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3621.781629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.956209] Lustre: lustre-OST0001: new disk, initializing [ 3621.964187] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3621.974986] Lustre: 126615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3623.402192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3623.420996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3623.531905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3628.532504] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3638.651609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3642.326500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4063.608234] Lustre: 133045:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4063.628142] Lustre: 133045:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4063.642911] Lustre: 133045:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4063.657186] Lustre: 133045:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4063.664308] Lustre: 133045:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4063.671332] Lustre: 133045:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4346.155697] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:09:02 (1786979342) [ 4363.758486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.762786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.777648] Lustre: Skipped 7 previous similar messages [ 4363.803888] Lustre: Skipped 15 previous similar messages [ 4368.265462] Lustre: server umount lustre-MDT0000 complete [ 4368.873583] LustreError: 124175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.899247] LustreError: 124175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4372.133533] LustreError: 125746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979371 with bad export cookie 11070382058446074822 [ 4372.136288] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.149895] LustreError: 125746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4373.987762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.011633] Lustre: Skipped 1 previous similar message [ 4374.021264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4378.929156] Lustre: server umount lustre-MDT0001 complete [ 4397.023132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4397.184095] Lustre: server umount lustre-OST0000 complete [ 4406.772846] Lustre: server umount lustre-OST0001 complete [ 4412.572993] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 4422.600227] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4458.544028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.770826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4458.802530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4458.909476] Lustre: lustre-MDT0000: new disk, initializing [ 4459.002305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4459.008526] Lustre: Skipped 5 previous similar messages [ 4459.022760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4463.030446] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4472.761523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.860841] Lustre: 136098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4472.906059] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4472.911728] Lustre: Skipped 1 previous similar message [ 4473.023309] Lustre: lustre-MDT0001: new disk, initializing [ 4473.184420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4473.211160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4478.100703] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4488.156451] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4494.091264] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4501.272380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.561640] Lustre: lustre-OST0000: new disk, initializing [ 4501.564125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4501.570099] Lustre: 138066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4503.461237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4503.472832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4503.559963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4508.125516] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4518.040448] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.282974] Lustre: lustre-MDD0000: changelog on [ 4531.142314] Lustre: lustre-MDD0001: changelog on [ 6409.903217] Lustre: lustre-MDD0001: changelog off [ 6411.946231] Lustre: 136106:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6412.721763] Lustre: lustre-MDD0000: changelog off [ 6423.521947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6423.523385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.528561] LustreError: Skipped 1 previous similar message [ 6423.529846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.554437] Lustre: Skipped 2 previous similar messages [ 6423.562115] Lustre: Skipped 3 previous similar messages [ 6424.765353] Lustre: server umount lustre-OST0000 complete [ 6429.153772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.160407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6429.171304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6429.188663] Lustre: Skipped 4 previous similar messages [ 6434.272501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6434.285535] Lustre: Skipped 1 previous similar message [ 6434.721361] Lustre: server umount lustre-MDT0000 complete [ 6438.951544] LustreError: 136093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981437 with bad export cookie 11070382058446850191 [ 6438.957177] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6438.977185] LustreError: 136093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6439.348028] Lustre: server umount lustre-MDT0001 complete [ 6453.101627] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 6457.437645] Key type lgssc unregistered [ 6458.157394] LNet: 147199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6458.182618] LNetError: 147199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6458.237915] LNet: Removed LNI 192.168.204.119@tcp [ 6459.843407] Key type .llcrypt unregistered [ 6459.848668] Key type ._llcrypt unregistered [ 6483.880747] Key type ._llcrypt registered [ 6483.882996] Key type .llcrypt registered [ 6484.027332] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 6499.493424] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 6500.446976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6500.564828] alg: No test for adler32 (adler32-zlib) [ 6501.652688] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 6502.014943] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 6503.735209] Key type lgssc registered [ 6505.040344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6541.455821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6541.488484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.768783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6542.811461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6542.894410] Lustre: lustre-MDT0000: new disk, initializing [ 6543.054482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.073597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6547.366857] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6558.686785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.910061] Lustre: 150400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6558.986975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6558.991978] Lustre: Skipped 1 previous similar message [ 6559.085148] Lustre: lustre-MDT0001: new disk, initializing [ 6559.249840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6559.314907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6559.333813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6564.495405] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6574.368022] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6580.205365] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6587.046461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.269427] Lustre: lustre-OST0000: new disk, initializing [ 6587.273459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6587.280668] Lustre: 152369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6587.380359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.113991] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6597.228981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6597.246852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6597.430450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6604.567697] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6610.876467] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_module obdecho/obdecho [ 6612.901291] LustreError: 153371:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6612.911755] LustreError: 153371:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6614.938531] LustreError: 153467:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6630.888484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6630.913854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.948168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6633.939418] Lustre: server umount lustre-MDT0000 complete [ 6637.840871] LustreError: 150393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981636 with bad export cookie 14100005241729084293 [ 6637.860393] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6637.860480] LustreError: 150393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6638.393698] Lustre: server umount lustre-MDT0001 complete [ 6653.206907] Lustre: server umount lustre-OST0000 complete [ 6674.559822] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 6686.133527] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 6723.846456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6724.038645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6724.062217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6724.130090] Lustre: lustre-MDT0000: new disk, initializing [ 6724.187694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6724.202735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6729.205528] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6740.436693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.657362] Lustre: 157186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6740.737110] LustreError: 157197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.749774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6740.751760] LustreError: 157197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6740.765132] Lustre: Skipped 1 previous similar message [ 6740.820724] Lustre: lustre-MDT0001: new disk, initializing [ 6740.930175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.984984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6741.009094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6746.857505] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6757.375917] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6764.376947] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6772.452599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6772.841063] Lustre: lustre-OST0000: new disk, initializing [ 6772.855155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6772.874233] Lustre: 159157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6773.063443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6779.327637] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6780.465157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6780.479871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6780.602293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6788.972178] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.215186] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6794.573825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6800.703659] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6801.069801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6806.510088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6806.511339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6806.520131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6806.529306] Lustre: Skipped 3 previous similar messages [ 6806.554895] Lustre: Skipped 4 previous similar messages [ 6809.767562] Lustre: server umount lustre-OST0000 complete [ 6816.740285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.757076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6816.765893] Lustre: Skipped 1 previous similar message [ 6820.437696] Lustre: server umount lustre-MDT0000 complete [ 6821.859879] LustreError: 157194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.898898] LustreError: 157194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6824.672660] LustreError: 157176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981823 with bad export cookie 14100005241729085252 [ 6824.676182] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.681707] LustreError: 157176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6824.923867] Lustre: server umount lustre-MDT0001 complete [ 6828.972395] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:50:25 (1786981825) [ 6830.691243] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6832.923263] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:50:29 (1786981829) [ 6841.931958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.488027] LustreError: 161572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.555412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.685287] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6847.969829] LustreError: 161573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.100496] LustreError: 161572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.514400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6862.090545] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6870.264428] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6877.997112] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6885.722571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6886.057232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6886.070567] Lustre: Skipped 1 previous similar message [ 6895.500733] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6903.229551] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6909.392456] LustreError: 161568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.561052] Lustre: server umount lustre-MDT0000 complete [ 6911.969812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.984923] Lustre: Skipped 2 previous similar messages [ 6913.541753] LustreError: 161553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981912 with bad export cookie 14100005241729086134 [ 6913.546685] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.557889] LustreError: 161553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6914.494900] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 6917.092170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.123442] Lustre: Skipped 1 previous similar message [ 6919.572981] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 6919.584993] Lustre: Skipped 2 previous similar messages [ 6920.097395] Lustre: server umount lustre-MDT0001 complete [ 6926.550664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6940.257788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.518639] LustreError: 165412:0:(llog.c:1655:llog_backup()) MGC192.168.204.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6940.534052] Lustre: 165412:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6943.713164] LustreError: 165417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6943.744678] LustreError: 165417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6979.555839] LustreError: 165417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.581428] LustreError: 165417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7001.569920] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7001.583596] Lustre: 165412:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7042.599211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.520230] LustreError: 165435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7045.540315] LustreError: 165435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 7047.398080] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7047.666945] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7056.334819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.592984] LustreError: 166158:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7058.602666] LustreError: 166158:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ca44e95c x1873785913472768/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786982057 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7058.621292] LustreError: 166158:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7058.912902] LustreError: 148623:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000604d7eb9 x1873785913477760/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7063.795378] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7064.582704] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7072.394934] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7073.795581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7079.165668] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7080.930859] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7100.388125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7100.394804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.408706] LustreError: Skipped 1 previous similar message [ 7100.411170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7100.419693] Lustre: Skipped 1 previous similar message [ 7100.441372] Lustre: Skipped 2 previous similar messages [ 7104.821242] Lustre: server umount lustre-MDT0000 complete [ 7109.152747] LustreError: 161552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982108 with bad export cookie 14100005241729092588 [ 7109.158619] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.168936] LustreError: 161552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7109.727803] Lustre: server umount lustre-MDT0001 complete [ 7125.027310] Lustre: server umount lustre-OST0000 complete [ 7149.704225] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 7161.041784] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7201.527912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.709456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7201.755514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7201.852165] Lustre: lustre-MDT0000: new disk, initializing [ 7201.953791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.957305] Lustre: Skipped 1 previous similar message [ 7201.971921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7206.518985] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7216.682204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.772584] Lustre: 170518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7217.007967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7221.153515] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7229.553566] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7235.520091] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7241.763599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7241.988936] Lustre: lustre-OST0000: new disk, initializing [ 7241.991226] Lustre: Skipped 1 previous similar message [ 7241.997590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7242.001432] Lustre: Skipped 2 previous similar messages [ 7242.005425] Lustre: 172488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7247.852851] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7248.486852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7248.519925] Lustre: Skipped 1 previous similar message [ 7248.526926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7248.708584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7260.141667] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7264.911894] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7265.131964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7269.783710] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7269.961519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7274.465884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.476718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.479215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.486241] Lustre: Skipped 2 previous similar messages [ 7274.503300] LustreError: Skipped 1 previous similar message [ 7277.933695] Lustre: server umount lustre-OST0000 complete [ 7283.680569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7287.934472] Lustre: server umount lustre-MDT0000 complete [ 7289.825882] LustreError: 170525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.846181] LustreError: 170525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7291.801985] LustreError: 170510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982290 with bad export cookie 14100005241729109423 [ 7291.814244] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.817326] LustreError: 170510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7292.157950] Lustre: server umount lustre-MDT0001 complete [ 7301.137583] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:58:17 (1786982297) [ 7306.743732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.753702] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7318.864261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.022177] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7331.641557] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7338.086173] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7346.348745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7354.468814] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7361.801151] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7376.184147] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:59:33 (1786982373) [ 7379.942619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7379.950182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.950191] Lustre: Skipped 2 previous similar messages [ 7379.954000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7379.986112] Lustre: Skipped 5 previous similar messages [ 7385.785915] Lustre: server umount lustre-OST0000 complete [ 7395.953275] Lustre: server umount lustre-MDT0000 complete [ 7399.719149] LustreError: 174877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982398 with bad export cookie 14100005241729110298 [ 7399.724253] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7399.729558] LustreError: 174877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7400.199226] Lustre: server umount lustre-MDT0001 complete [ 7412.238941] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 7415.152920] Key type lgssc unregistered [ 7415.429290] LNet: 178628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7415.442380] LNetError: 178628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7415.467856] LNet: Removed LNI 192.168.204.119@tcp [ 7416.247549] Key type .llcrypt unregistered [ 7416.250111] Key type ._llcrypt unregistered [ 7437.265283] Key type ._llcrypt registered [ 7437.268072] Key type .llcrypt registered [ 7437.409393] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7438.183802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7438.201725] alg: No test for adler32 (adler32-zlib) [ 7439.500292] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 7439.777377] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 7441.495288] Key type lgssc registered [ 7443.317156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7451.460516] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7451.472574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.975830] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7472.096233] LustreError: 179878:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 7477.215510] LustreError: 179878:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7477.222046] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7477.225770] LustreError: MGC192.168.204.119@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7477.242241] LustreError: 179878:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7477.256873] LustreError: 179878:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7477.262282] LustreError: 179878:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7477.349801] Lustre: server umount lustre-OST0000 complete [ 7477.353500] LustreError: 179878:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7485.860321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.342330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.171232] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7497.700172] LustreError: 180389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.728374] LustreError: 180389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7500.845797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.220971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7505.986091] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7512.740510] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7518.458995] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7525.729939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7526.002549] LustreError: 182204:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7526.013195] Lustre: 182204:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.119@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7526.186788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7533.043816] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7540.691775] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7547.361208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7547.369870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.392434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.883043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.902316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.924255] Lustre: Skipped 1 previous similar message [ 7547.938563] Lustre: Skipped 1 previous similar message [ 7549.395437] Lustre: server umount lustre-MDT0000 complete [ 7552.953560] LustreError: 180369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982551 with bad export cookie 1805722100668315031 [ 7552.964239] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7552.965385] LustreError: 180369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7552.995185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7552.998116] LustreError: 181628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.018672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7553.379259] Lustre: server umount lustre-MDT0001 complete [ 7559.926732] Lustre: server umount lustre-OST0000 complete [ 7570.659290] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 7581.110545] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7622.557614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.731202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7622.754615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7622.819505] Lustre: lustre-MDT0000: new disk, initializing [ 7622.878512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.896465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7627.145650] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7637.621792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.758852] Lustre: 185729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7637.807912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7637.812460] Lustre: Skipped 1 previous similar message [ 7637.881450] Lustre: lustre-MDT0001: new disk, initializing [ 7638.004288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.052385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7638.069309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7643.377382] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7653.118353] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7659.029906] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7666.058180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7666.332822] Lustre: lustre-OST0000: new disk, initializing [ 7666.336836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7666.347704] Lustre: 187695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7666.434333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7673.259672] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7676.491312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7676.499599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7676.616516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7682.592858] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7687.469511] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7687.702654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7692.731238] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7693.016617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7697.379784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7697.380365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.390770] LustreError: Skipped 1 previous similar message [ 7697.393034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7697.415667] Lustre: Skipped 1 previous similar message [ 7702.502924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7702.515352] Lustre: Skipped 2 previous similar messages [ 7709.151164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7709.321620] Lustre: server umount lustre-OST0000 complete [ 7714.785306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7714.801668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.815826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7714.820956] Lustre: Skipped 2 previous similar messages [ 7719.596263] Lustre: server umount lustre-MDT0000 complete [ 7722.976593] LustreError: 185741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.996811] LustreError: 185741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7723.088924] LustreError: 185722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982721 with bad export cookie 1805722100668315745 [ 7723.094467] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.095882] LustreError: 185722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7723.487878] Lustre: server umount lustre-MDT0001 complete [ 7731.682630] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:05:28 (1786982728) [ 7748.206282] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 7751.010044] Key type lgssc unregistered [ 7751.476620] LNet: 190646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7751.484622] LNetError: 190646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7751.513376] LNet: Removed LNI 192.168.204.119@tcp [ 7752.351790] Key type .llcrypt unregistered [ 7752.354235] Key type ._llcrypt unregistered [ 7771.903670] Key type ._llcrypt registered [ 7771.908114] Key type .llcrypt registered [ 7772.037592] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7772.852953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7772.881989] alg: No test for adler32 (adler32-zlib) [ 7774.158937] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 7774.370820] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 7776.063190] Key type lgssc registered [ 7777.486829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7784.570482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7784.587541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7785.902038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.653493] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7795.173369] LustreError: 191864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.185116] LustreError: 191864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7799.462381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.012332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7805.361840] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7813.471031] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7819.746837] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7827.559974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7827.744184] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7827.747595] LustreError: 193727:0:(llog.c:1696:llog_backup()) MGC192.168.204.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7827.755966] Lustre: 193727:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.119@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7827.914170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7834.567176] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7841.977294] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7844.438738] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 7844.448244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.459281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7845.857151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7845.864966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7845.886919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7849.444972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7849.445961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.462226] Lustre: Skipped 1 previous similar message [ 7850.809216] Lustre: server umount lustre-MDT0000 complete [ 7854.314161] LustreError: 191844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982853 with bad export cookie 9248029536002513397 [ 7854.322228] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7854.328791] LustreError: 191844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7854.563847] LustreError: 191858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.568558] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.604958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7854.614400] Lustre: Skipped 1 previous similar message [ 7854.647319] LustreError: 192593:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a3f632f4 x1873787248209664/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7854.761663] Lustre: server umount lustre-MDT0001 complete [ 7861.010154] Lustre: server umount lustre-OST0000 complete [ 7872.049832] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 7883.078876] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7925.885843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.288317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7926.326704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7926.447157] Lustre: lustre-MDT0000: new disk, initializing [ 7926.571579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7926.598702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7931.645914] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7942.916996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.080354] Lustre: 197254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7943.161368] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7943.168461] Lustre: Skipped 1 previous similar message [ 7943.175563] LustreError: 198035:0:(llog.c:1696:llog_backup()) MGC192.168.204.119@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7943.184301] LustreError: 198035:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7943.188727] Lustre: 198035:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.119@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7943.248160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7943.251929] Lustre: Skipped 1 previous similar message [ 7943.321121] Lustre: lustre-MDT0001: new disk, initializing [ 7943.432254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7943.476701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7943.493720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7948.428816] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7958.930225] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7965.307481] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7972.201375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7972.519879] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7972.522951] Lustre: Skipped 1 previous similar message [ 7972.547713] LustreError: 199198:0:(llog.c:1696:llog_backup()) MGC192.168.204.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7972.554644] LustreError: 199198:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7972.559870] Lustre: 199198:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.119@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7972.571858] Lustre: 199198:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7972.592162] Lustre: lustre-OST0000: new disk, initializing [ 7972.599532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7972.611951] Lustre: 199221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7972.714529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.047030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7978.065589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7978.185398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7979.869445] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7989.607332] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7994.859503] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7995.108860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8000.595606] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8000.778151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8004.069827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8004.078676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.085242] LustreError: Skipped 1 previous similar message [ 8004.089047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.105782] Lustre: Skipped 1 previous similar message [ 8004.132077] Lustre: Skipped 1 previous similar message [ 8009.722158] Lustre: server umount lustre-OST0000 complete [ 8014.305742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.321877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.326166] Lustre: Skipped 2 previous similar messages [ 8020.563898] Lustre: server umount lustre-MDT0000 complete [ 8024.546871] LustreError: 197260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.569976] LustreError: 197260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8024.958487] LustreError: 197244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983023 with bad export cookie 9248029536002514118 [ 8024.964485] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.974274] LustreError: 197244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8025.612798] Lustre: server umount lustre-MDT0001 complete [ 8035.392577] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:10:32 (1786983032) [ 8053.831259] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8065.304805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.947471] LustreError: 202348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.047231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8071.123898] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8071.141832] LustreError: 202349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.171287] LustreError: 202349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8080.353861] LustreError: 202348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.372759] LustreError: 202348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8080.562896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.142389] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8089.784573] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8097.264946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.939976] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8115.735633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8115.906650] Lustre: lustre-OST0001: new disk, initializing [ 8115.911615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8115.922860] Lustre: 204657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8116.096243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8116.103875] Lustre: Skipped 2 previous similar messages [ 8122.809709] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8122.997171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8123.017728] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8123.239071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8133.005764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8140.611328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8142.554775] Lustre: DEBUG MARKER: ADD OST3 [ 8147.984348] loop: module loaded [ 8148.097992] loop0: detected capacity change from 0 to 204800000 [ 8148.511285] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8148.545921] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8148.567441] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8155.248298] loop0: detected capacity change from 0 to 204800000 [ 8155.902138] Lustre: DEBUG MARKER: STOP OST3 [ 8160.994944] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.185950] Lustre: lustre-OST0003: new disk, initializing [ 8161.191306] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8161.204320] Lustre: 206692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8162.842434] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8162.972163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8162.980635] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8163.027672] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8164.217971] Lustre: Failing over lustre-OST0003 [ 8164.363031] Lustre: server umount lustre-OST0003 complete [ 8164.895547] LustreError: 206842:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8166.646363] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8168.363797] LustreError: 204664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.430682] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8168.435465] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.455578] Lustre: Skipped 2 previous similar messages [ 8176.479659] Lustre: DEBUG MARKER: START OST3 again [ 8183.760754] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8184.010211] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8184.014402] Lustre: Skipped 1 previous similar message [ 8184.031743] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8185.577726] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8185.686432] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8185.688506] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8190.031747] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8210.658237] Lustre: Failing over lustre-OST0003 [ 8210.793814] Lustre: server umount lustre-OST0003 complete [ 8211.424800] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8211.433573] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.448893] LustreError: 203837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8211.471484] LustreError: 203837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8229.345285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8229.350988] LustreError: Skipped 1 previous similar message [ 8229.355593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8229.362331] Lustre: Skipped 2 previous similar messages [ 8236.618111] Lustre: server umount lustre-MDT0000 complete [ 8240.511191] LustreError: 202329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983239 with bad export cookie 9248029536002515000 [ 8240.524154] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8240.532085] LustreError: 202329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8247.028611] Lustre: server umount lustre-MDT0001 complete [ 8250.935681] Lustre: server umount lustre-OST0000 complete [ 8254.886371] Lustre: server umount lustre-OST0001 complete [ 8262.690632] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 8271.583889] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8308.213738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.448341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8308.490405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8308.552877] Lustre: lustre-MDT0000: new disk, initializing [ 8308.633172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8313.066415] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8323.998635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.135988] Lustre: 212123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8324.399708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8324.408222] Lustre: Skipped 1 previous similar message [ 8324.480200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8329.825958] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8339.169103] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8344.487843] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.869506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.160590] Lustre: lustre-OST0000: new disk, initializing [ 8351.163134] Lustre: Skipped 1 previous similar message [ 8351.166277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8351.171164] Lustre: Skipped 2 previous similar messages [ 8351.175521] Lustre: 214090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8352.656484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8352.670274] Lustre: Skipped 1 previous similar message [ 8352.682540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8352.809780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8357.748359] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8366.295248] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.628107] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8370.843766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8375.680939] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8375.878412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8378.338663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8378.342035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.356068] LustreError: Skipped 1 previous similar message [ 8378.359114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8378.371444] Lustre: Skipped 8 previous similar messages [ 8378.385392] Lustre: Skipped 9 previous similar messages [ 8383.912877] Lustre: server umount lustre-OST0000 complete [ 8394.216398] Lustre: server umount lustre-MDT0000 complete [ 8397.439457] LustreError: 212113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983396 with bad export cookie 9248029536002517296 [ 8397.447549] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8397.459783] LustreError: 212113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8397.827822] Lustre: server umount lustre-MDT0001 complete [ 8401.335360] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:16:38 (1786983398) [ 8420.026714] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 8430.917471] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8466.401663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.676771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8466.709747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8466.791970] Lustre: lustre-MDT0000: new disk, initializing [ 8466.877947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8471.576731] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8482.775768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8482.940369] Lustre: 218700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8483.281693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8488.634200] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8498.434574] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8503.902472] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8509.775347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8510.102493] Lustre: 220667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8511.777986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8516.186610] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8525.417200] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8530.231950] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8530.404877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8534.831255] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8535.059469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8537.587695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.588155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8537.591900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.591910] Lustre: Skipped 6 previous similar messages [ 8537.594707] Lustre: Skipped 3 previous similar messages [ 8537.607526] LustreError: Skipped 1 previous similar message [ 8551.392108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8551.628348] Lustre: server umount lustre-OST0000 complete [ 8561.993797] Lustre: server umount lustre-MDT0000 complete [ 8563.170134] LustreError: 220675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8563.205910] LustreError: 220675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8566.303662] LustreError: 218693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983565 with bad export cookie 9248029536002518178 [ 8566.314146] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8566.768810] Lustre: server umount lustre-MDT0001 complete [ 8576.940078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.954576] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8591.011218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.560213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8591.565216] Lustre: Skipped 5 previous similar messages [ 8596.910826] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8605.335305] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8611.545349] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8620.473951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.158493] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8638.013064] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8648.674245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.413403] Lustre: server umount lustre-OST0000 complete [ 8663.282847] Lustre: server umount lustre-MDT0000 complete [ 8666.777834] LustreError: 222870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983665 with bad export cookie 9248029536002519060 [ 8666.782634] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.789774] LustreError: 222870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8667.326174] Lustre: server umount lustre-MDT0001 complete [ 8776.204123] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 8779.260474] Key type lgssc unregistered [ 8779.621904] LNet: 226555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8779.633041] LNetError: 226555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8779.654129] LNet: Removed LNI 192.168.204.119@tcp [ 8780.814345] Key type .llcrypt unregistered [ 8780.815985] Key type ._llcrypt unregistered [ 8807.055698] Key type ._llcrypt registered [ 8807.057665] Key type .llcrypt registered [ 8807.168987] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 8823.860781] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8824.908848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8825.024897] alg: No test for adler32 (adler32-zlib) [ 8826.128842] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8826.329222] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 8828.015434] Key type lgssc registered [ 8829.543488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8871.356385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8871.405547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.792522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8872.821861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8872.995351] Lustre: lustre-MDT0000: new disk, initializing [ 8873.065276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8873.099146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8878.898070] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8891.631593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.734153] Lustre: 229913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8891.802923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8891.808295] Lustre: Skipped 1 previous similar message [ 8891.890220] Lustre: lustre-MDT0001: new disk, initializing [ 8892.041696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8892.076381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8892.084381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8897.855287] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8908.746270] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8915.624299] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8923.026435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.521608] Lustre: lustre-OST0000: new disk, initializing [ 8923.530717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8923.551661] Lustre: 231881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8923.675439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.337436] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8931.966424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8931.990448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8932.140236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8941.143214] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8945.616501] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8945.891513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8950.481859] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8950.732831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8952.812234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.812595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8952.826019] LustreError: Skipped 1 previous similar message [ 8952.830349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8952.835653] Lustre: Skipped 1 previous similar message [ 8957.928085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8957.947030] Lustre: Skipped 1 previous similar message [ 8963.053140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8963.068750] Lustre: Skipped 1 previous similar message [ 8967.135152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8967.360305] Lustre: server umount lustre-OST0000 complete [ 8973.280871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8973.310903] Lustre: Skipped 1 previous similar message [ 8973.322143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8973.332420] Lustre: Skipped 1 previous similar message [ 8977.954695] Lustre: server umount lustre-MDT0000 complete [ 8981.877833] LustreError: 229905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983980 with bad export cookie 7816948765271955302 [ 8981.883664] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8981.895553] LustreError: 229905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8982.233962] Lustre: server umount lustre-MDT0001 complete [ 9003.418361] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9015.486653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.118787] LustreError: 234817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9016.140399] LustreError: 234817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9016.222768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.408727] LustreError: 234818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.800137] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9026.531707] LustreError: 234817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9031.442219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.648783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9031.907484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9037.241221] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9040.684493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9047.705756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.097702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9054.265945] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9064.184463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9064.373891] Lustre: lustre-OST0001: new disk, initializing [ 9064.379951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9064.387458] Lustre: 237125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9064.489750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9072.183478] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9074.891610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9074.908747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9075.092943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9083.932987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9091.054442] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9108.607143] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:28:25 (1786984105) [ 9111.041323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9111.050101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9111.074160] Lustre: Skipped 4 previous similar messages [ 9111.095484] Lustre: Skipped 3 previous similar messages [ 9125.347182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9125.635142] Lustre: server umount lustre-MDT0000 complete [ 9126.378724] LustreError: 234813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.405101] LustreError: 234813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9129.112736] LustreError: 234799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984127 with bad export cookie 7816948765271956184 [ 9129.122471] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9129.126206] LustreError: 234799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9129.511636] Lustre: server umount lustre-MDT0001 complete [ 9142.319564] Lustre: server umount lustre-OST0000 complete [ 9157.332916] Lustre: server umount lustre-OST0001 complete [ 9165.296641] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 9179.338801] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9221.828837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9222.147442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9222.181256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9222.294990] Lustre: lustre-MDT0000: new disk, initializing [ 9222.432183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.475532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9227.796331] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9239.956280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.083994] Lustre: 241076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9240.143105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9240.148193] Lustre: Skipped 1 previous similar message [ 9240.238860] Lustre: lustre-MDT0001: new disk, initializing [ 9240.466938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9240.486555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9246.292891] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9257.570409] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9264.247656] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9272.411265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9272.789252] Lustre: lustre-OST0000: new disk, initializing [ 9272.796551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9272.809688] Lustre: 243044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9273.966938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9273.997841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9274.139100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9280.519949] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9291.653731] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.643928] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9298.107421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9305.470860] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9305.873357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9310.181332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9310.194122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9310.199201] LustreError: Skipped 1 previous similar message [ 9310.199541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9310.199544] Lustre: Skipped 8 previous similar messages [ 9310.244738] Lustre: Skipped 1 previous similar message [ 9314.654501] Lustre: server umount lustre-OST0000 complete [ 9320.416649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.439636] Lustre: Skipped 1 previous similar message [ 9326.607451] Lustre: server umount lustre-MDT0000 complete [ 9330.660387] LustreError: 241084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9330.724412] LustreError: 241084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9331.566760] LustreError: 241070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984330 with bad export cookie 7816948765271958319 [ 9331.570719] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.582380] LustreError: 241070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9332.234828] Lustre: server umount lustre-MDT0001 complete [ 9343.422105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.957013] LustreError: 245268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.061615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.070517] Lustre: Skipped 2 previous similar messages [ 9350.401638] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9360.301842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.948740] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9375.304893] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9383.103761] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9391.352853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.605438] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9409.899595] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9560.033252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9560.061742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.098665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9560.107152] Lustre: Skipped 5 previous similar messages [ 9562.089045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9564.780307] Lustre: server umount lustre-OST0000 complete [ 9570.784878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9575.517184] Lustre: server umount lustre-MDT0000 complete [ 9577.442498] LustreError: 245996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9577.468524] LustreError: 245996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9579.074200] LustreError: 248094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984577 with bad export cookie 7816948765271959201 [ 9579.076600] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9579.086603] LustreError: 248094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9579.322471] Lustre: server umount lustre-MDT0001 complete [ 9678.039738] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 9680.589735] Key type lgssc unregistered [ 9680.868137] LNet: 248881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9680.887183] LNetError: 248881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9680.900272] LNet: Removed LNI 192.168.204.119@tcp [ 9682.056303] Key type .llcrypt unregistered [ 9682.059139] Key type ._llcrypt unregistered [ 9708.099077] Key type ._llcrypt registered [ 9708.101554] Key type .llcrypt registered [ 9708.199566] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 9725.402633] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9726.821927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9726.868844] alg: No test for adler32 (adler32-zlib) [ 9727.899359] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 9728.205627] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 9729.879448] Key type lgssc registered [ 9731.271550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9782.987144] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9795.733701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9795.781771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.122584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9797.161131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9797.251299] Lustre: lustre-MDT0000: new disk, initializing [ 9797.398126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.435493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9803.521925] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9816.221853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.347940] Lustre: 252957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9816.430896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9816.435678] Lustre: Skipped 1 previous similar message [ 9816.529839] Lustre: lustre-MDT0001: new disk, initializing [ 9816.666726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9816.714694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9816.732378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9821.828152] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9827.701846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9835.549908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9835.847502] Lustre: lustre-OST0000: new disk, initializing [ 9835.851703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9835.858339] Lustre: 254587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9835.984492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.749087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9837.759693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9837.859060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9843.453853] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9853.944604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.118851] Lustre: lustre-OST0001: new disk, initializing [ 9854.123096] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9854.129214] Lustre: 255459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9854.268678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9860.263629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9860.271873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9860.416110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9861.364373] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9872.447431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9883.886957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9896.419746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.442395] Lustre: Skipped 3 previous similar messages [ 9896.443208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9896.455866] Lustre: Skipped 3 previous similar messages [ 9901.543226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9901.550128] Lustre: Skipped 3 previous similar messages [ 9902.643567] Lustre: server umount lustre-MDT0000 complete [ 9907.182348] LustreError: 252950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984906 with bad export cookie 13518562260766952948 [ 9907.194065] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9907.209996] LustreError: 252950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9907.802823] Lustre: server umount lustre-MDT0001 complete [ 9923.061423] Lustre: server umount lustre-OST0000 complete [ 9927.135926] Lustre: 250463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786984910/real 1786984910] req@ffff942cfbb2f480 x1873789297181312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786984926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9927.180309] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.179240] Lustre: server umount lustre-OST0001 complete [ 9944.274229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9954.590808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9964.218663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9973.361342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9994.956299] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10006.015592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.038054] Lustre: lustre-MDT0000: reset Object Index mappings [10006.564809] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.582263] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10006.651522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10011.728917] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10012.129462] LustreError: 260472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10016.227216] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.356613] LustreError: 260472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.794602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.828205] Lustre: lustre-MDT0001: reset Object Index mappings [10022.380537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10027.667418] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10031.122629] Lustre: 261614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10038.283304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10038.857821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10045.722121] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10048.103285] LustreError: 261970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.199061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10053.700121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10059.297120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10061.296781] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10068.869968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10072.776575] Lustre: 263485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10078.667223] Lustre: server umount lustre-MDT0000 complete [10078.688343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10078.703756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10078.727654] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.754769] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10081.951973] LustreError: 260452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985080 with bad export cookie 13518562260766959738 [10081.957164] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10081.965423] LustreError: 260452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10082.287632] Lustre: server umount lustre-MDT0001 complete [10096.075864] Lustre: server umount lustre-OST0000 complete [10100.194282] Lustre: 250464:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786985083/real 1786985083] req@ffff942cddab9180 x1873789297229440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786985099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10100.217439] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10100.228242] Lustre: Skipped 3 previous similar messages [10101.177826] Lustre: server umount lustre-OST0001 complete [10124.968988] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:45:21 (1786985121) [10143.729716] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [10155.496350] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10194.452871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.684291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10194.722359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10194.818465] Lustre: lustre-MDT0000: new disk, initializing [10194.972378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10194.983159] Lustre: Skipped 1 previous similar message [10195.020976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10200.271637] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10213.501872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.743564] Lustre: 267741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10213.872908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10213.882712] Lustre: Skipped 1 previous similar message [10214.042646] Lustre: lustre-MDT0001: new disk, initializing [10214.248226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10214.263145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10219.112866] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10229.471911] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10236.297971] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10244.281914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10244.684908] Lustre: lustre-OST0000: new disk, initializing [10244.690350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10244.711652] Lustre: 269706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10246.241213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10246.251663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10246.422069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10251.980411] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10261.936831] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10267.098859] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10267.331098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10272.381723] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10272.666805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10277.347124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.356215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10277.373870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10280.998235] Lustre: server umount lustre-OST0000 complete [10286.050890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10286.069521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.093852] Lustre: Skipped 1 previous similar message [10286.098088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.107167] Lustre: Skipped 1 previous similar message [10292.710527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10292.725079] Lustre: Skipped 3 previous similar messages [10299.360402] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10299.537412] Lustre: server umount lustre-MDT0000 complete [10302.944848] LustreError: 267751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.973614] LustreError: 267751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10304.310309] LustreError: 267733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985303 with bad export cookie 13518562260766961831 [10304.315636] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10304.326528] LustreError: 267733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10304.768269] Lustre: server umount lustre-MDT0001 complete [10326.737584] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10338.144976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10338.614016] LustreError: 272645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.698979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10338.707594] Lustre: Skipped 2 previous similar messages [10342.780332] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10350.881041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.474765] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10359.888465] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10367.448748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10374.298977] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10383.469876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.683775] Lustre: lustre-OST0001: new disk, initializing [10383.688317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10383.703816] Lustre: 274957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10385.020782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10385.030612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10385.214713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10390.732196] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10400.345469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10403.783478] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10409.058873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10409.190996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10416.101622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10416.106451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10416.122752] Lustre: Skipped 3 previous similar messages [10416.131113] Lustre: Skipped 3 previous similar messages [10422.191023] Lustre: server umount lustre-MDT0000 complete [10426.352657] LustreError: 272640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10426.377132] LustreError: 272640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10426.403818] LustreError: 272627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985425 with bad export cookie 13518562260766962713 [10426.417844] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.423047] LustreError: 272627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10426.791225] Lustre: server umount lustre-MDT0001 complete [10440.819259] Lustre: server umount lustre-OST0000 complete [10455.225351] Lustre: server umount lustre-OST0001 complete [10472.820273] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10483.311488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.867167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10483.879389] Lustre: Skipped 3 previous similar messages [10488.569856] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10496.530618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10501.523203] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10504.340653] Lustre: 279151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10511.378881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10519.335549] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10528.250269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10528.507374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10533.924575] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10535.895612] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10543.526849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10547.315706] Lustre: 281028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10552.523491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10552.559691] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10561.736705] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:52:38 (1786985558) [10578.914456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10578.922920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.935316] Lustre: Skipped 1 previous similar message [10578.940299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.945796] Lustre: Skipped 6 previous similar messages [10583.186705] Lustre: server umount lustre-MDT0000 complete [10585.059109] LustreError: 278007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.080606] LustreError: 278007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10586.848402] LustreError: 277993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985585 with bad export cookie 13518562260766969034 [10586.850803] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.863361] LustreError: 277993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10593.340669] Lustre: server umount lustre-MDT0001 complete [10603.939521] Lustre: server umount lustre-OST0000 complete [10607.675907] Lustre: server umount lustre-OST0001 complete [10613.652877] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [10624.745789] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10675.828463] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10685.987181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.241915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10686.291312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10686.363527] Lustre: lustre-MDT0000: new disk, initializing [10686.471679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10690.831504] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10703.151414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10703.295863] Lustre: 285267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10703.743213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10708.991150] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10722.535024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.961796] Lustre: 286902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10725.052281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10730.423303] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10741.958179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10742.108492] Lustre: 287774:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10742.217370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10742.228037] Lustre: Skipped 6 previous similar messages [10743.625876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10743.644922] Lustre: Skipped 1 previous similar message [10743.809055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10749.815943] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10760.955555] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10765.675677] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10765.681677] Lustre: Skipped 1 previous similar message [10783.156581] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:56:19 (1786985779) [10785.765612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10785.770826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10785.786782] Lustre: Skipped 5 previous similar messages [10785.793200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10785.797691] Lustre: Skipped 8 previous similar messages [10787.711934] Lustre: server umount lustre-MDT0000 complete [10791.443265] LustreError: 285261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985790 with bad export cookie 13518562260766976902 [10791.461929] LustreError: 285261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10791.464246] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10791.799495] Lustre: server umount lustre-MDT0001 complete [10806.716887] Lustre: server umount lustre-OST0000 complete [10821.063305] Lustre: server umount lustre-OST0001 complete [10830.085941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.377625] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10843.689920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.221400] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10857.999994] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10863.743685] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10870.141078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10875.702381] LustreError: 291849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.721075] LustreError: 291849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10876.993395] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10884.986284] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10905.343149] loop1: detected capacity change from 0 to 204800000 [10905.551860] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10905.565839] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10905.591144] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10911.438829] loop1: detected capacity change from 0 to 204800000 [10917.918281] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.043928] LustreError: 290006:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13889: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10918.057148] LustreError: 290006:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13889 [10918.061067] LustreError: 290006:0:(mgs_handler.c:517:mgs_target_reg()) fs13889-MDT0001: Error checking target index: rc = -13 [10918.067781] LustreError: fs13889-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10918.073676] LustreError: 293743:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10918.077816] LustreError: 293743:0:(tgt_mount.c:2129:server_put_super()) no obd fs13889-MDT0001 [10918.082512] LustreError: 293743:0:(tgt_mount.c:116:server_deregister_mount()) fs13889-MDT0001 not registered [10918.122700] Lustre: server umount fs13889-MDT0001 complete [10918.127921] LustreError: 293743:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10926.562990] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.656412] Lustre: Setting parameter fs13889-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13889-MDT0001 [10926.702414] Lustre: srv-fs13889-MDT0001: No data found on store. Initialize space. [10926.708162] Lustre: Skipped 4 previous similar messages [10926.798368] Lustre: fs13889-MDT0001: new disk, initializing [10926.803762] Lustre: Skipped 3 previous similar messages [10926.935611] Lustre: srv-fs13889-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10926.955994] Lustre: 294210:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13889-MDT0001-mdtlov: can't create new llog: rc = -115 [10926.969919] LustreError: 294210:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13889-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10931.248802] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10938.027983] Lustre: server umount fs13889-MDT0001 complete [10952.049144] loop1: detected capacity change from 0 to 204800000 [10954.754443] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10957.507342] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.893979] Lustre: 295775:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13889-MDT0001-mdtlov: can't create new llog: rc = -115 [10957.902855] LustreError: 295775:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13889-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10962.787334] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10967.170089] Lustre: server umount fs13889-MDT0001 complete [10977.251893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10977.266149] LustreError: Skipped 1 previous similar message [10977.281113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10977.305459] Lustre: Skipped 3 previous similar messages [10977.311887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.075789] Lustre: server umount lustre-MDT0000 complete [10991.166140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.317459] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10995.145520] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10996.719037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11002.344064] loop1: detected capacity change from 0 to 204800000 [11007.776421] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.346287] Lustre: 298220:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13889-MDT0001-mdtlov: can't create new llog: rc = -115 [11008.358673] LustreError: 298220:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13889-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11014.002976] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11018.813983] Lustre: server umount fs13889-MDT0001 complete [11048.870567] Lustre: server umount lustre-OST0000 complete [11053.026057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11058.792352] Lustre: server umount lustre-MDT0000 complete [11063.231502] LustreError: 291874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986062 with bad export cookie 13518562260766979583 [11063.240417] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11063.242577] LustreError: 291874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11063.763720] Lustre: server umount lustre-MDT0001 complete [11075.661186] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11078.625378] Key type lgssc unregistered [11078.902652] LNet: 300578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11078.912525] LNetError: 300578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11078.937227] LNet: Removed LNI 192.168.204.119@tcp [11079.879442] Key type .llcrypt unregistered [11079.881687] Key type ._llcrypt unregistered [11100.702358] Key type ._llcrypt registered [11100.705202] Key type .llcrypt registered [11100.820813] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11102.146688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11102.174790] alg: No test for adler32 (adler32-zlib) [11103.320616] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11103.603460] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11105.287267] Key type lgssc registered [11106.438852] Lustre: Echo OBD driver; http://www.lustre.org/ [11114.732772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11114.785705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.528420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11121.293889] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11126.752990] LustreError: 301850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.761044] LustreError: 301850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11129.799611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.313825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11135.524809] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11142.811230] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11149.442379] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11156.130694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11156.536543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11161.640044] LustreError: 303932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.643567] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11166.691251] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.777835] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11171.823387] LustreError: 303682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.910839] LustreError: 303682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.935607] LustreError: 303682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11186.150076] LustreError: 303683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.182918] LustreError: 303683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11192.277202] loop1: detected capacity change from 0 to 204800000 [11192.486282] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11192.499164] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11192.514876] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11198.117644] loop1: detected capacity change from 0 to 204800000 [11202.963206] LustreError: 303683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.982226] LustreError: 303683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11203.615992] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11203.819587] Lustre: fs8500-OST0000: new disk, initializing [11203.824280] Lustre: srv-fs8500-OST0000: No data found on store. Initialize space. [11203.833552] Lustre: 305583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11203.932115] Lustre: fs8500-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11211.031829] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11220.060633] Lustre: server umount fs8500-OST0000 complete [11231.350558] loop1: detected capacity change from 0 to 204800000 [11237.356482] LustreError: 306446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11237.372706] LustreError: 306446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11237.976901] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11238.389328] Lustre: fs8500-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11242.478527] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11247.205619] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11256.954281] loop2: detected capacity change from 0 to 204800000 [11257.193772] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11257.221992] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11257.260752] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11257.858511] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11263.461592] loop2: detected capacity change from 0 to 204800000 [11270.562078] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11270.667155] LustreError: fs8500-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11270.680680] LustreError: fs8500-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11270.691359] LustreError: 308625:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11270.700782] LustreError: 308625:0:(tgt_mount.c:2129:server_put_super()) no obd fs8500-OST0001 [11270.708626] LustreError: 308625:0:(tgt_mount.c:116:server_deregister_mount()) fs8500-OST0001 not registered [11270.750675] Lustre: server umount fs8500-OST0001 complete [11270.752930] LustreError: 308625:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11282.310531] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11282.472106] Lustre: fs8500-OST0001: new disk, initializing [11282.476878] Lustre: srv-fs8500-OST0001: No data found on store. Initialize space. [11282.484469] Lustre: 309177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11282.555467] Lustre: fs8500-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11288.757975] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11296.411727] Lustre: server umount fs8500-OST0000 complete [11303.931250] LustreError: 303682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.965899] LustreError: 303682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11305.006534] Lustre: server umount fs8500-OST0001 complete [11316.583797] loop1: detected capacity change from 0 to 204800000 [11321.782331] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.101849] Lustre: fs8500-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11328.961941] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11336.415329] loop2: detected capacity change from 0 to 204800000 [11341.968446] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.112521] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11357.461492] Lustre: server umount fs8500-OST0000 complete [11367.336571] Lustre: server umount fs8500-OST0001 complete [11380.710055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.720175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11380.722657] Lustre: Skipped 1 previous similar message [11380.745143] Lustre: Skipped 1 previous similar message [11385.830882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11385.844471] Lustre: Skipped 1 previous similar message [11386.902627] Lustre: server umount lustre-OST0000 complete [11391.458473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11391.466982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.477508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.068514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.072850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11397.742412] Lustre: server umount lustre-MDT0000 complete [11401.874481] LustreError: 301831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986400 with bad export cookie 11028446933052984408 [11401.875851] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11401.884061] LustreError: 301831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11402.213676] Lustre: server umount lustre-MDT0001 complete [11414.483563] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11418.523551] Key type lgssc unregistered [11418.868693] LNet: 315103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11418.880357] LNetError: 315103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11418.920946] LNet: Removed LNI 192.168.204.119@tcp [11420.337260] Key type .llcrypt unregistered [11420.342682] Key type ._llcrypt unregistered [11453.941670] Key type ._llcrypt registered [11453.947040] Key type .llcrypt registered [11454.074789] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [11475.513131] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11477.812909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11477.877832] alg: No test for adler32 (adler32-zlib) [11479.115045] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11479.633600] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11481.400314] Key type lgssc registered [11483.291821] Lustre: Echo OBD driver; http://www.lustre.org/ [11528.278728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11528.293096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11529.557197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11529.591588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11529.678603] Lustre: lustre-MDT0000: new disk, initializing [11529.793144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11529.821914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11534.658858] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11547.213822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11547.338341] Lustre: 318965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11547.394801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11547.398050] Lustre: Skipped 1 previous similar message [11547.465722] Lustre: lustre-MDT0001: new disk, initializing [11547.552189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11547.596277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11547.607092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11552.668468] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11562.384444] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11567.815365] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11575.556129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11575.895435] Lustre: lustre-OST0000: new disk, initializing [11575.904404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11575.916417] Lustre: 320933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11576.074076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11582.007763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11582.019869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11582.173329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11583.276537] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11592.540570] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11594.444555] Lustre: Mounted lustre-client - version 2.17.57_1_gad8061f [11595.364086] Lustre: Unmounted lustre-client [11604.967136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.970133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11604.998671] Lustre: Skipped 1 previous similar message [11605.024213] Lustre: Skipped 1 previous similar message [11610.084371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11610.098499] Lustre: Skipped 1 previous similar message [11610.842980] Lustre: server umount lustre-OST0000 complete [11615.201802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.203758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11615.219908] Lustre: Skipped 1 previous similar message [11615.232874] Lustre: Skipped 1 previous similar message [11620.322306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.327372] Lustre: Skipped 1 previous similar message [11621.222446] Lustre: server umount lustre-MDT0000 complete [11626.237219] LustreError: 318955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986625 with bad export cookie 13076680344593026509 [11626.259975] LustreError: 318955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11626.263104] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11626.592413] Lustre: server umount lustre-MDT0001 complete [11640.052718] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11643.431417] Key type lgssc unregistered [11648.929453] LNet: 323100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11648.946909] LNetError: 323100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11723.750215] LNet: Removed LNI 192.168.204.119@tcp [11724.690358] Key type .llcrypt unregistered [11724.694845] Key type ._llcrypt unregistered [11742.088983] Key type ._llcrypt registered [11742.090822] Key type .llcrypt registered [11742.236686] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [11760.465629] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11761.457465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11761.642399] alg: No test for adler32 (adler32-zlib) [11763.144569] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11763.639614] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11765.423443] Key type lgssc registered [11767.130738] Lustre: Echo OBD driver; http://www.lustre.org/ [11799.621251] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:13:16 (1786986796) [11806.596700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11806.642323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.887873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11807.911640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11807.965679] Lustre: lustre-MDT0000: new disk, initializing [11808.035767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.055799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11813.011551] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11822.879730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.029217] Lustre: 326325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11823.089402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11823.094328] Lustre: Skipped 1 previous similar message [11823.219996] Lustre: lustre-MDT0001: new disk, initializing [11823.371819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11823.457612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11823.472074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11828.495918] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11837.104632] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11842.083328] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11848.381692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.759713] Lustre: lustre-OST0000: new disk, initializing [11848.765776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11848.777429] Lustre: 328293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11848.942122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11854.433899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11854.456118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11854.546839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11856.907349] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11867.309314] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11885.542747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11885.549530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.563069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11887.853711] Lustre: server umount lustre-OST0000 complete [11895.265966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11895.271769] LustreError: Skipped 1 previous similar message [11895.280176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11895.296339] Lustre: Skipped 1 previous similar message [11895.304385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11895.311895] Lustre: Skipped 1 previous similar message [11898.018366] Lustre: server umount lustre-MDT0000 complete [11902.814409] LustreError: 326315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986901 with bad export cookie 5567779131177061727 [11902.815464] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11902.836761] LustreError: 326315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11903.192271] Lustre: server umount lustre-MDT0001 complete [11915.088608] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11918.366671] Key type lgssc unregistered [11918.689535] LNet: 330203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11918.694046] LNetError: 330203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11918.705648] LNet: Removed LNI 192.168.204.119@tcp [11919.619411] Key type .llcrypt unregistered [11919.621405] Key type ._llcrypt unregistered [11942.469650] Key type ._llcrypt registered [11942.472370] Key type .llcrypt registered [11942.595036] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11943.696486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11943.854470] alg: No test for adler32 (adler32-zlib) [11944.892914] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11945.180108] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11946.927534] Key type lgssc registered [11948.394593] Lustre: Echo OBD driver; http://www.lustre.org/ [11956.736865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11956.764831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.335853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11963.709820] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11968.486470] LustreError: 331477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.500634] LustreError: 331477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11972.899699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11973.376672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11978.369339] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11986.754849] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11992.645222] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12000.082956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12000.485089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12007.276621] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12018.354531] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12028.929587] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12030.172546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12034.531612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12034.536322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12034.550541] Lustre: Skipped 2 previous similar messages [12034.560666] Lustre: Skipped 2 previous similar messages [12036.541131] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.19@tcp (stopping) [12039.648983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12039.666286] Lustre: Skipped 2 previous similar messages [12040.938371] Lustre: server umount lustre-MDT0000 complete [12041.620451] LustreError: 333478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.769490] LustreError: 331478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.787597] LustreError: 331478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12044.905107] LustreError: 331458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987043 with bad export cookie 1556404946066737082 [12044.909585] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12044.913174] LustreError: 331458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12046.770685] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [12049.889845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.890177] LustreError: 331474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12049.916195] LustreError: 331474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12051.204309] Lustre: server umount lustre-MDT0001 complete [12055.072509] Lustre: server umount lustre-OST0000 complete [12063.585162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.027420] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12070.968643] Lustre: 335661:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.119@tcp [12073.426038] Lustre: server umount MGS complete [12081.596917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.157377] LustreError: 336234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.250990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12087.190317] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12091.799599] LustreError: 336230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12091.816770] LustreError: 336230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12096.439942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.770686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12101.051871] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12109.725808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.148597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.976673] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12124.400281] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12126.052952] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12131.010190] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12132.528409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12137.960109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.967522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12137.971286] Lustre: Skipped 1 previous similar message [12137.992214] Lustre: Skipped 2 previous similar messages [12143.285186] Lustre: server umount lustre-OST0000 complete [12148.196812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12148.197527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12148.203811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12148.203821] Lustre: Skipped 2 previous similar messages [12148.241890] Lustre: Skipped 1 previous similar message [12153.636124] Lustre: server umount lustre-MDT0000 complete [12157.478909] LustreError: 336214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987156 with bad export cookie 1556404946066737915 [12157.488247] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12157.495437] LustreError: 336214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12157.842401] Lustre: server umount lustre-MDT0001 complete [12169.358873] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12172.104775] Key type lgssc unregistered [12172.421690] LNet: 339673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.428762] LNetError: 339673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12172.456562] LNet: Removed LNI 192.168.204.119@tcp [12173.399541] Key type .llcrypt unregistered [12173.401598] Key type ._llcrypt unregistered [12193.009180] Key type ._llcrypt registered [12193.015667] Key type .llcrypt registered [12193.114423] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [12209.373767] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12210.743350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12210.770395] alg: No test for adler32 (adler32-zlib) [12211.742046] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [12212.037165] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [12213.679189] Key type lgssc registered [12215.053564] Lustre: Echo OBD driver; http://www.lustre.org/ [12254.077319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12254.096277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.334888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12255.388703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12255.537547] Lustre: lustre-MDT0000: new disk, initializing [12255.673350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12255.714937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12260.676469] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12273.324132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12273.582718] Lustre: 342835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12273.727163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12273.739352] Lustre: Skipped 1 previous similar message [12273.846752] Lustre: lustre-MDT0001: new disk, initializing [12273.956806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12274.018327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12274.035410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12279.379724] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12290.103153] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12296.147148] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12303.046644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12303.324037] Lustre: lustre-OST0000: new disk, initializing [12303.327996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12303.333772] Lustre: 344805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12303.410941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12309.218289] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12312.754275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12312.786096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12313.026479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12319.879488] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12325.458299] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12325.833996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12330.347265] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12330.536541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12333.541872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.547911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12333.561781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12333.571766] Lustre: Skipped 1 previous similar message [12338.756442] Lustre: server umount lustre-OST0000 complete [12343.776908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12343.781685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12343.786702] Lustre: Skipped 2 previous similar messages [12343.809883] Lustre: Skipped 1 previous similar message [12348.896978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12348.916455] Lustre: Skipped 1 previous similar message [12349.663803] Lustre: server umount lustre-MDT0000 complete [12353.939403] LustreError: 342830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987352 with bad export cookie 12920467942804545688 [12353.957245] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12353.958647] LustreError: 342830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12354.376771] Lustre: server umount lustre-MDT0001 complete [12366.430713] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:22:42 (1786987362) [12384.150676] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [12393.076509] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12428.988757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.465380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12429.507957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12429.599630] Lustre: lustre-MDT0000: new disk, initializing [12429.759089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12429.780780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12435.096653] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12445.156719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12445.285820] Lustre: 349567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12445.341763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12445.347683] Lustre: Skipped 1 previous similar message [12445.357860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12445.435385] Lustre: lustre-MDT0001: new disk, initializing [12445.536051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12445.582484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12445.598529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12450.858636] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12462.801319] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12469.274676] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12476.534173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12476.788441] Lustre: lustre-OST0000: new disk, initializing [12476.792549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12476.797748] Lustre: 351536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12476.903720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12483.169977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12483.181653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12483.328529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12484.386106] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12494.487522] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12529.635081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12529.646072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12529.679719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12534.793546] Lustre: server umount lustre-OST0000 complete [12539.313060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12539.323236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12539.342793] Lustre: Skipped 2 previous similar messages [12539.358974] Lustre: Skipped 2 previous similar messages [12545.629454] Lustre: server umount lustre-MDT0000 complete [12549.345059] LustreError: 349559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987548 with bad export cookie 12920467942804546563 [12549.349633] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12549.351362] LustreError: 349559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12549.742971] Lustre: server umount lustre-MDT0001 complete [12561.263875] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12564.661802] Key type lgssc unregistered [12565.076497] LNet: 353716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12565.085622] LNetError: 353716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12565.104058] LNet: Removed LNI 192.168.204.119@tcp [12566.121996] Key type .llcrypt unregistered [12566.132873] Key type ._llcrypt unregistered [12656.266317] Key type ._llcrypt registered [12656.268841] Key type .llcrypt registered [12656.411590] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12656.887334] Key type .llcrypt unregistered [12656.888920] Key type ._llcrypt unregistered [12734.220342] Key type ._llcrypt registered [12734.223921] Key type .llcrypt registered [12734.335785] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12734.625364] Key type .llcrypt unregistered [12734.632113] Key type ._llcrypt unregistered [12760.262753] Key type ._llcrypt registered [12760.264988] Key type .llcrypt registered [12760.365809] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12760.632633] Key type .llcrypt unregistered [12760.636709] Key type ._llcrypt unregistered [12786.928915] Key type ._llcrypt registered [12786.933039] Key type .llcrypt registered [12787.045828] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12787.554482] Key type .llcrypt unregistered [12787.557688] Key type ._llcrypt unregistered [12864.987455] Key type ._llcrypt registered [12864.988806] Key type .llcrypt registered [12865.127194] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12865.521334] Key type .llcrypt unregistered [12865.523110] Key type ._llcrypt unregistered [12896.835583] Key type ._llcrypt registered [12896.838159] Key type .llcrypt registered [12896.968069] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12897.245382] Key type .llcrypt unregistered [12897.249703] Key type ._llcrypt unregistered [12923.467385] Key type ._llcrypt registered [12923.476765] Key type .llcrypt registered [12923.595626] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12923.881607] Key type .llcrypt unregistered [12923.885354] Key type ._llcrypt unregistered [12953.052744] Key type ._llcrypt registered [12953.059411] Key type .llcrypt registered [12953.205539] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12953.548570] Key type .llcrypt unregistered [12953.560788] Key type ._llcrypt unregistered [12987.688951] Key type ._llcrypt registered [12987.691493] Key type .llcrypt registered [12987.826923] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12988.174632] Key type .llcrypt unregistered [12988.178260] Key type ._llcrypt unregistered [13023.494315] Key type ._llcrypt registered [13023.498433] Key type .llcrypt registered [13023.689395] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13024.090495] Key type .llcrypt unregistered [13024.092258] Key type ._llcrypt unregistered [13087.133208] Key type ._llcrypt registered [13087.137327] Key type .llcrypt registered [13087.278388] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13087.566307] Key type .llcrypt unregistered [13087.568565] Key type ._llcrypt unregistered [13120.230487] Key type ._llcrypt registered [13120.237918] Key type .llcrypt registered [13120.359155] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13121.048421] Key type .llcrypt unregistered [13121.052575] Key type ._llcrypt unregistered [13150.919959] Key type ._llcrypt registered [13150.928197] Key type .llcrypt registered [13151.048865] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13151.287633] Key type .llcrypt unregistered [13151.291465] Key type ._llcrypt unregistered [13203.418338] Key type ._llcrypt registered [13203.420635] Key type .llcrypt registered [13203.509773] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [13219.062445] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [13220.483682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13220.567152] alg: No test for adler32 (adler32-zlib) [13221.538250] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [13221.755395] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [13223.407385] Key type lgssc registered [13224.478392] Lustre: Echo OBD driver; http://www.lustre.org/ [13262.199620] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13262.247356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13263.508365] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13263.555578] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13263.685797] Lustre: tst250fs-MDT0000: new disk, initializing [13263.751558] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13263.764377] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13269.468081] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13282.316798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13282.503437] Lustre: 374017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13282.547695] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13282.550963] Lustre: Skipped 1 previous similar message [13282.619740] Lustre: tst250fs-MDT0001: new disk, initializing [13282.711147] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13282.755381] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13282.772148] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13287.871052] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13293.694329] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13300.830359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13301.344255] Lustre: tst250fs-OST0000: new disk, initializing [13301.357625] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13301.376732] Lustre: 375648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13301.572926] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13307.505228] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13307.519860] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13307.637818] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13308.094586] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13318.770448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13318.946477] Lustre: tst250fs-OST0001: new disk, initializing [13318.951314] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13318.959063] Lustre: 376519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13319.069893] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13324.923313] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13328.460599] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13328.482735] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13328.577852] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13329.633183] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13332.230783] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13332.236472] Lustre: Skipped 2 previous similar messages [13349.347536] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.351915] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13349.352633] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13349.353930] Lustre: Skipped 1 previous similar message [13354.471072] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13354.476528] Lustre: Skipped 6 previous similar messages [13354.827828] Lustre: server umount tst250fs-MDT0000 complete [13358.107231] LustreError: 374008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988356 with bad export cookie 5956027907119568587 [13358.117614] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13358.126238] LustreError: 374008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13358.650108] Lustre: server umount tst250fs-MDT0001 complete [13372.214794] Lustre: server umount tst250fs-OST0000 complete [13375.469147] Lustre: 372240:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786988358/real 1786988358] req@ffff942cde26ce00 x1873792959838848/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786988374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13375.503890] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13375.533698] Lustre: Skipped 2 previous similar messages [13376.248645] Lustre: server umount tst250fs-OST0001 complete [13396.674448] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [13405.872651] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [13438.919800] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:40:35 (1786988435) [13441.191336] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13465.201473] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [13476.647632] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [13509.018928] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13194 sec ========= 13:41:45 (1786988505) [13511.555257] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:41:47 (1786988507) === [13515.639904] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:41:51 (1786988511) === [13559.577526] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13562.375932] Key type lgssc unregistered [13562.745824] LNet: 385648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13562.753948] LNetError: 385648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13562.776240] LNet: Removed LNI 192.168.204.119@tcp [13563.791505] Key type .llcrypt unregistered [13563.795938] Key type ._llcrypt unregistered