[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481926175 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2496MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002239] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004027] kvm-guest: setup PV IPIs [ 0.007377] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011063] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.023216] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028335] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032116] *** VALIDATE tmpfs *** [ 0.033353] *** VALIDATE proc *** [ 0.034235] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038111] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.043954] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.045203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046763] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054292] rcu: Hierarchical SRCU implementation. [ 0.056502] smp: Bringing up secondary CPUs ... [ 0.057631] x86: Booting SMP configuration: [ 0.058037] .... node #0, CPUs: #1 #2 #3 [ 0.061428] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.147291] node 0 deferred pages initialised in 81ms [ 0.149112] devtmpfs: initialized [ 0.150274] x86/mm: Memory block size: 128MB [ 0.152952] gcov: version magic: 0x41383552 [ 0.154277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156284] pinctrl core: initialized pinctrl subsystem [ 0.157333] [ 0.157893] ************************************************************* [ 0.158016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159104] ** ** [ 0.160016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161256] ** ** [ 0.162016] ** This means that this kernel is built to expose internal ** [ 0.163017] ** IOMMU data structures, which may compromise security on ** [ 0.164013] ** your system. ** [ 0.165015] ** ** [ 0.166014] ** If you see this message and you are not debugging the ** [ 0.167016] ** kernel, report this immediately to your vendor! ** [ 0.168012] ** ** [ 0.169012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170010] ************************************************************* [ 0.171730] NET: Registered protocol family 16 [ 0.172465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175551] cpuidle: using governor menu [ 0.178029] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179459] PCI: Using configuration type 1 for base access [ 0.180138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187074] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189202] cryptd: max_cpu_qlen set to 1000 [ 0.191206] ACPI: Added _OSI(Module Device) [ 0.192012] ACPI: Added _OSI(Processor Device) [ 0.193013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197986] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201127] ACPI: Interpreter enabled [ 0.202061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203015] ACPI: Using IOAPIC for interrupt routing [ 0.204102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214230] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219442] acpiphp: Slot [2] registered [ 0.220089] acpiphp: Slot [5] registered [ 0.220946] acpiphp: Slot [6] registered [ 0.221070] acpiphp: Slot [7] registered [ 0.221976] acpiphp: Slot [8] registered [ 0.222145] acpiphp: Slot [9] registered [ 0.223133] acpiphp: Slot [10] registered [ 0.224205] acpiphp: Slot [3] registered [ 0.225114] acpiphp: Slot [4] registered [ 0.226109] acpiphp: Slot [11] registered [ 0.227106] acpiphp: Slot [12] registered [ 0.228127] acpiphp: Slot [13] registered [ 0.229122] acpiphp: Slot [14] registered [ 0.230107] acpiphp: Slot [15] registered [ 0.231092] acpiphp: Slot [16] registered [ 0.232107] acpiphp: Slot [17] registered [ 0.233107] acpiphp: Slot [18] registered [ 0.234136] acpiphp: Slot [19] registered [ 0.235104] acpiphp: Slot [20] registered [ 0.236103] acpiphp: Slot [21] registered [ 0.237108] acpiphp: Slot [22] registered [ 0.238142] acpiphp: Slot [23] registered [ 0.239128] acpiphp: Slot [24] registered [ 0.240127] acpiphp: Slot [25] registered [ 0.241134] acpiphp: Slot [26] registered [ 0.242117] acpiphp: Slot [27] registered [ 0.243139] acpiphp: Slot [28] registered [ 0.244110] acpiphp: Slot [29] registered [ 0.245115] acpiphp: Slot [30] registered [ 0.246100] acpiphp: Slot [31] registered [ 0.247068] PCI host bridge to bus 0000:00 [ 0.248023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.249025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.251026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.253031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.254166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255998] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.258333] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.267058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.268023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.269012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.270012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.271586] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.272777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.273039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.274977] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.277021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.286026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.288874] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.293054] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.296027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.299023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.306034] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.314299] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.317026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.320027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.327028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.333972] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.336018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.339019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.347017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.353779] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.367019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.376019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.396019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.407709] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.417017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.430018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.464020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.484038] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.495020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.507017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.523016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.533827] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.536384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.539371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.541374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.543186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.550045] iommu: Default domain type: Passthrough [ 0.552083] SCSI subsystem initialized [ 0.553139] ACPI: bus type USB registered [ 0.555139] usbcore: registered new interface driver usbfs [ 0.557092] usbcore: registered new interface driver hub [ 0.559089] usbcore: registered new device driver usb [ 0.561363] pps_core: LinuxPPS API ver. 1 registered [ 0.564026] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.566078] PTP clock support registered [ 0.570088] EDAC MC: Ver: 3.0.0 [ 0.571354] PCI: Using ACPI for IRQ routing [ 0.572806] NetLabel: Initializing [ 0.573013] NetLabel: domain hash size = 128 [ 0.574012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.575098] NetLabel: unlabeled traffic allowed by default [ 0.578152] vgaarb: loaded [ 0.580248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.581009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.586319] clocksource: Switched to clocksource kvm-clock [ 0.695432] VFS: Disk quotas dquot_6.6.0 [ 0.697578] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.700935] *** VALIDATE ramfs *** [ 0.702434] *** VALIDATE hugetlbfs *** [ 0.704205] pnp: PnP ACPI init [ 0.706189] pnp: PnP ACPI: found 6 devices [ 0.726033] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.729229] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.731619] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.733865] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.736429] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.739811] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.743897] NET: Registered protocol family 2 [ 0.747103] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.752770] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.757733] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.764365] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.768732] TCP: Hash tables configured (established 65536 bind 65536) [ 0.772774] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.775989] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.778957] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.781977] NET: Registered protocol family 1 [ 0.784475] RPC: Registered named UNIX socket transport module. [ 0.786551] RPC: Registered udp transport module. [ 0.788413] RPC: Registered tcp transport module. [ 0.790245] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.792493] NET: Registered protocol family 44 [ 0.794074] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.796108] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.798285] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.800771] PCI: CLS 0 bytes, default 64 [ 0.802611] Unpacking initramfs... [ 2.213950] debug: unmapping init [mem 0xffff9b8a7cc54000-0xffff9b8a7ffbffff] [ 2.216850] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.218572] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.221185] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.703200] Initialise system trusted keyrings [ 2.704766] Key type blacklist registered [ 2.706525] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.715483] zbud: loaded [ 2.718229] *** VALIDATE nfs *** [ 2.719444] *** VALIDATE nfs4 *** [ 2.721023] pstore: using deflate compression [ 2.726320] Platform Keyring initialized [ 2.817351] NET: Registered protocol family 38 [ 2.818687] Key type asymmetric registered [ 2.819621] Asymmetric key parser 'x509' registered [ 2.820776] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.823953] io scheduler mq-deadline registered [ 2.825914] io scheduler kyber registered [ 2.827779] io scheduler bfq registered [ 2.830848] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.833824] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.836764] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.839795] ACPI: Power Button [PWRF] [ 2.844974] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.851444] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.863912] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.871679] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.885525] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.914263] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.943262] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.947845] Non-volatile memory driver v1.3 [ 2.949636] Linux agpgart interface v0.103 [ 2.981667] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 2.983907] vda: detected capacity change from 0 to 69632000 [ 2.999322] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.002599] vdb: detected capacity change from 0 to 1073741824 [ 3.017146] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.020503] vdc: detected capacity change from 0 to 2621440000 [ 3.035667] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.037837] vdd: detected capacity change from 0 to 2621440000 [ 3.053237] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.056514] vde: detected capacity change from 0 to 4294967296 [ 3.073345] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.076044] vdf: detected capacity change from 0 to 4294967296 [ 3.081640] libphy: Fixed MDIO Bus: probed [ 3.093785] usbcore: registered new interface driver usbserial_generic [ 3.096323] usbserial: USB Serial support registered for generic [ 3.099215] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.105050] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.107427] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.110427] mousedev: PS/2 mouse device common for all mice [ 3.113577] rtc_cmos 00:05: RTC can wake from S4 [ 3.116235] rtc_cmos 00:05: registered as rtc0 [ 3.118074] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.118238] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.120891] intel_pstate: CPU model not supported [ 3.127157] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.127616] hid: raw HID events driver (C) Jiri Kosina [ 3.132327] usbcore: registered new interface driver usbhid [ 3.134348] usbhid: USB HID core driver [ 3.136029] drop_monitor: Initializing network drop monitor service [ 3.136064] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.138862] Initializing XFRM netlink socket [ 3.144408] NET: Registered protocol family 10 [ 3.147084] Segment Routing with IPv6 [ 3.148631] NET: Registered protocol family 17 [ 3.150885] mpls_gso: MPLS GSO support [ 3.157498] RAS: Correctable Errors collector initialized. [ 3.159861] AVX version of gcm_enc/dec engaged. [ 3.161772] AES CTR mode by8 optimization enabled [ 3.221583] sched_clock: Marking stable (3221564057, 0)->(4345547827, -1123983770) [ 3.223938] registered taskstats version 1 [ 3.227120] Loading compiled-in X.509 certificates [ 3.230260] zswap: loaded using pool lzo/zbud [ 3.249932] Key type big_key registered [ 3.259096] Key type encrypted registered [ 3.260237] ima: No TPM chip found, activating TPM-bypass! [ 3.261436] ima: Allocated hash algorithm: sha1 [ 3.262654] ima: No architecture policies found [ 3.263722] evm: Initialising EVM extended attributes: [ 3.264848] evm: security.selinux [ 3.265806] evm: security.ima [ 3.266861] evm: security.capability [ 3.267936] evm: HMAC attrs: 0x1 [ 3.270319] rtc_cmos 00:05: setting system clock to 2026-06-03 05:13:30 UTC (1780463610) [ 3.275490] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 3.277621] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 3.285045] Write protecting the kernel read-only data: 28672k [ 3.287041] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 3.288962] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 3.314821] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.320406] systemd[1]: Detected virtualization kvm. [ 3.321501] systemd[1]: Detected architecture x86-64. [ 3.323124] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.352685] systemd[1]: No hostname configured. [ 3.353940] systemd[1]: Set hostname to . [ 3.355304] random: systemd: uninitialized urandom read (16 bytes read) [ 3.356752] systemd[1]: Initializing machine ID from random generator. [ 3.470320] random: systemd: uninitialized urandom read (16 bytes read) [ 3.473469] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.480972] random: systemd: uninitialized urandom read (16 bytes read) [ 3.484612] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.490355] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.011390] device-mapper: uevent: version 1.0.3 [ 4.012970] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.632112] virtio_net virtio0 ens2: renamed from eth0 [ 4.650169] random: fast init done [ 4.710945] scsi host0: ata_piix [ 4.714883] scsi host1: ata_piix [ 4.716407] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.718276] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.192522] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.772691] random: crng init done [ 9.774183] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.951441] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.040312] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.351710] SELinux: Disabled at runtime. [ 11.415051] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.427748] systemd[1]: Detected virtualization kvm. [ 11.433844] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.137787] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.144529] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.157704] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.166763] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.174477] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.192577] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.206579] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ 13.324332] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.782815] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.044694] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.105119] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.737482] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.890740] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 21.093825] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 22.024515] NFS: Registering the id_resolver key type [ 22.028560] Key type id_resolver registered [ 22.035505] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg260-server login: [ 79.391082] hrtimer: interrupt took 9965168 ns [ 83.655577] libcfs: loading out-of-tree module taints kernel. [ 83.710606] Key type ._llcrypt registered [ 83.715187] Key type .llcrypt registered [ 83.813346] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 104.256744] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 106.577643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.589821] alg: No test for adler32 (adler32-zlib) [ 107.929560] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 108.823681] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 110.535158] Key type lgssc registered [ 112.843786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.523309] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.411424] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 195.558850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 195.611495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.981451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.071145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.252537] Lustre: lustre-MDT0000: new disk, initializing [ 197.404442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.425445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.097707] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.492172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.656139] Lustre: 6499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.725777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.732870] Lustre: Skipped 1 previous similar message [ 216.900657] Lustre: lustre-MDT0001: new disk, initializing [ 217.018874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.070130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.081053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.925403] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.747578] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.976503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.265885] Lustre: lustre-OST0000: new disk, initializing [ 240.272128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.277700] Lustre: 8404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.343923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.655509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.665705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.722286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.725653] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.445445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.636284] Lustre: lustre-OST0001: new disk, initializing [ 261.638714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.646988] Lustre: 9460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.730313] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.279158] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.413934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.432334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.511575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 280.938548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 288.242313] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.183470] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.491619] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing yml_node [ 303.967382] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 306.589480] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 309.157805] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 310.722851] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jun 3 01:18:37 EDT 2026 [ 327.566318] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 329.155275] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.056379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.068936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.092887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.591660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.591989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.613944] Lustre: Skipped 2 previous similar messages [ 350.868510] Lustre: server umount lustre-MDT0000 complete [ 354.584207] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780463961 with bad export cookie 10034905915020430877 [ 354.587721] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.598454] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 356.840677] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.865104] Lustre: Skipped 1 previous similar message [ 356.865861] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.866043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.866046] Lustre: Skipped 3 previous similar messages [ 356.924164] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 359.840472] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.841265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.871598] Lustre: Skipped 2 previous similar messages [ 361.196955] Lustre: server umount lustre-MDT0001 complete [ 365.073498] Lustre: server umount lustre-OST0000 complete [ 368.903708] Lustre: server umount lustre-OST0001 complete [ 376.129708] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 386.992600] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 428.576059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.922685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.965872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 429.103317] Lustre: lustre-MDT0000: new disk, initializing [ 429.238535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.258441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 434.187709] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 446.609626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.799873] Lustre: 14979:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 446.900522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 446.904677] Lustre: Skipped 1 previous similar message [ 447.027202] Lustre: lustre-MDT0001: new disk, initializing [ 447.156212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.222487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.241593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.782786] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 459.964781] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 463.628989] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 471.093351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.417773] Lustre: lustre-OST0000: new disk, initializing [ 471.422934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.433372] Lustre: 16720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.566722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.928485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 472.946795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.134686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.746024] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 486.825674] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 491.423524] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.704115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.222567] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.466130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.661773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.663153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.707777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.716758] Lustre: Skipped 1 previous similar message [ 504.571627] Lustre: server umount lustre-OST0000 complete [ 514.019449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.023511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.034155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.034165] Lustre: Skipped 2 previous similar messages [ 514.050063] Lustre: Skipped 2 previous similar messages [ 514.100587] LustreError: Skipped 1 previous similar message [ 515.493323] Lustre: server umount lustre-MDT0000 complete [ 519.138149] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.152933] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 519.698888] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464126 with bad export cookie 10034905915020432410 [ 519.704555] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.716515] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 520.081074] Lustre: server umount lustre-MDT0001 complete [ 526.003289] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:22:12 (1780464132) [ 545.921314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.371965] LustreError: 19447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.435026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.223469] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 551.921690] LustreError: 19447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.975345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.973285] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 570.069419] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 574.217615] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.519094] Lustre: Failing over lustre-MDT0000 [ 577.088159] Lustre: server umount lustre-MDT0000 complete [ 580.579728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.590917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.630308] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.649505] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 596.374156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.548832] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.303419] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 602.083948] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 602.087641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 602.117731] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 608.675652] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 612.320912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 612.331570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.344381] Lustre: Skipped 1 previous similar message [ 612.353184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.358755] Lustre: Skipped 2 previous similar messages [ 618.484848] Lustre: server umount lustre-MDT0000 complete [ 621.823201] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464229 with bad export cookie 10034905915020433824 [ 621.839341] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.350609] Lustre: server umount lustre-MDT0001 complete [ 629.961354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.466209] LustreError: 22444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.486314] LustreError: 22444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 630.538358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.543478] Lustre: Skipped 2 previous similar messages [ 634.432130] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 642.518631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.650660] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 651.581808] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 654.701672] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 656.362724] Lustre: Failing over lustre-MDT0000 [ 658.400452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 658.410937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.426273] Lustre: Skipped 1 previous similar message [ 658.632928] Lustre: server umount lustre-MDT0000 complete [ 665.568276] LustreError: 22444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.577259] LustreError: 22444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 675.817639] Lustre: 3650:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780464267/real 1780464267] req@ffff9b89c3817100 x1866951521653376/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780464283 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 675.841201] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.582020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.399351] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 687.330069] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 687.336132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 687.354902] Lustre: Skipped 1 previous similar message [ 687.369647] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 690.207548] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 698.024541] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 701.933735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.947892] Lustre: Skipped 3 previous similar messages [ 716.255312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 716.551819] Lustre: server umount lustre-MDT0000 complete [ 720.269032] LustreError: 22425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464327 with bad export cookie 10034905915020434762 [ 720.288406] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.290430] LustreError: 22425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 720.617848] Lustre: server umount lustre-MDT0001 complete [ 728.799483] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:25:35 (1780464335) [ 742.905889] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 753.003900] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 787.996060] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 796.693917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.959204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 796.997889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 797.089894] Lustre: lustre-MDT0000: new disk, initializing [ 797.180923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.184416] Lustre: Skipped 2 previous similar messages [ 797.192337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 800.683416] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 809.494803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.574226] Lustre: 28429:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 809.743805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 813.428430] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 820.074165] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 823.007637] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 829.999783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 830.256863] Lustre: 30167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 831.511272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 831.551610] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 831.603413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 831.613959] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 835.457782] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 841.284902] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 845.357038] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 862.176476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 862.182948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.192295] Lustre: Skipped 3 previous similar messages [ 862.201160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 862.208572] Lustre: Skipped 4 previous similar messages [ 865.492247] Lustre: server umount lustre-OST0000 complete [ 875.020813] Lustre: server umount lustre-MDT0000 complete [ 878.048205] LustreError: 28440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.068217] LustreError: 28440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 878.546437] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464485 with bad export cookie 10034905915020435168 [ 878.550327] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.554482] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 878.975290] Lustre: server umount lustre-MDT0001 complete [ 889.255685] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 891.679908] Key type lgssc unregistered [ 891.904352] LNet: 31995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.912348] LNetError: 31995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.966151] LNet: Removed LNI 192.168.202.160@tcp [ 893.692136] Key type .llcrypt unregistered [ 893.694542] Key type ._llcrypt unregistered [ 911.210461] Key type ._llcrypt registered [ 911.214374] Key type .llcrypt registered [ 911.275559] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 912.290697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.314921] alg: No test for adler32 (adler32-zlib) [ 913.386594] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 913.619353] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 915.263162] Key type lgssc registered [ 916.180155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.477476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.504488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.990829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.927920] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 934.370137] LustreError: 33372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.383777] LustreError: 33372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 934.555258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.771796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.509063] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 942.595983] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.198194] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 950.616933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 950.848747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.051626] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 959.041271] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 959.049313] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.292074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 980.902301] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:29:48 (1780464588) [ 982.188205] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 983.473159] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 988.283393] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:29:55 (1780464595) [ 994.864205] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:30:01 (1780464601) [ 995.821104] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1002.437438] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1002.444432] Lustre: Skipped 2 previous similar messages [ 1003.774181] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:30:10 (1780464610) [ 1006.814264] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1006.819775] Lustre: Skipped 3 previous similar messages [ 1020.796918] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1020.802137] Lustre: Skipped 6 previous similar messages [ 1023.072939] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:30:29 (1780464629) [ 1025.324408] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1028.251953] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1034.963732] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:30:42 (1780464642) [ 1036.940841] Lustre: Disabling parameter general.jobid_name= in log params [ 1036.942599] Lustre: Skipped 4 previous similar messages [ 1045.512305] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:30:52 (1780464652) [ 1056.229092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1056.231368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.232368] LustreError: Skipped 1 previous similar message [ 1056.233099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1056.253911] Lustre: Skipped 1 previous similar message [ 1061.344522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.349753] Lustre: Skipped 1 previous similar message [ 1066.479371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1066.489637] Lustre: Skipped 1 previous similar message [ 1070.047145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1070.209833] Lustre: server umount lustre-OST0000 complete [ 1073.120508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1073.125941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.131593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.134675] Lustre: Skipped 1 previous similar message [ 1076.705683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.324970] Lustre: server umount lustre-MDT0000 complete [ 1081.826520] LustreError: 33367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.833331] LustreError: 33367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1082.217376] LustreError: 33353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464689 with bad export cookie 6412812239666308646 [ 1082.218822] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1082.223938] LustreError: 33353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1082.426427] Lustre: server umount lustre-MDT0001 complete [ 1090.040615] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1091.850936] Key type lgssc unregistered [ 1092.047321] LNet: 40874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.051367] LNetError: 40874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.060818] LNet: Removed LNI 192.168.202.160@tcp [ 1092.627142] Key type .llcrypt unregistered [ 1092.629722] Key type ._llcrypt unregistered [ 1105.247141] Key type ._llcrypt registered [ 1105.248480] Key type .llcrypt registered [ 1105.321779] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 1116.776603] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1117.647656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.675793] alg: No test for adler32 (adler32-zlib) [ 1118.625936] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1118.789651] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1120.431528] Key type lgssc registered [ 1121.324529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.171770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.191326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.363873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1147.386728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1147.441493] Lustre: lustre-MDT0000: new disk, initializing [ 1147.498099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.512879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1149.949350] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1156.958853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.016760] Lustre: 43967:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1157.054106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1157.056736] Lustre: Skipped 1 previous similar message [ 1157.096626] Lustre: lustre-MDT0001: new disk, initializing [ 1157.154141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1157.184119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1157.194234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1159.671932] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1164.696628] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1166.977787] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1171.474099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1171.639460] Lustre: lustre-OST0000: new disk, initializing [ 1171.642572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1171.647863] Lustre: 45707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1171.720193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.911449] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1178.738716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1178.748828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1178.858104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1181.971773] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1184.995254] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1185.140635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1187.793737] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1187.946883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1189.347458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.347889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1189.363433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1194.469948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1194.473588] Lustre: Skipped 1 previous similar message [ 1195.253651] Lustre: server umount lustre-OST0000 complete [ 1198.048765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1198.062988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.070455] Lustre: Skipped 1 previous similar message [ 1198.073870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.075915] Lustre: Skipped 1 previous similar message [ 1199.586309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.823819] Lustre: server umount lustre-MDT0000 complete [ 1206.789635] LustreError: 43960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464814 with bad export cookie 15514384794966529952 [ 1206.791052] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.801518] LustreError: 43960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1207.116149] Lustre: server umount lustre-MDT0001 complete [ 1213.462056] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:33:40 (1780464820) [ 1218.291704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.638792] LustreError: 48018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.651861] LustreError: 48018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1218.699283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.577799] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1224.161508] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.807444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.987890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1230.212834] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1233.489423] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1235.793913] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1240.185266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1240.401509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.749606] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1248.849279] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1256.260932] LustreError: 50205:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1256.929298] Lustre: Modifying parameter general.timeout=1 in log params [ 1258.208469] Lustre: Modifying parameter general.timeout=3 in log params [ 1258.212273] Lustre: Skipped 1 previous similar message [ 1260.558835] Lustre: Modifying parameter general.timeout=7 in log params [ 1260.561770] Lustre: Skipped 3 previous similar messages [ 1265.090104] Lustre: Modifying parameter general.timeout=15 in log params [ 1265.092621] Lustre: Skipped 7 previous similar messages [ 1273.383443] Lustre: Modifying parameter general.timeout=29 in log params [ 1273.386440] Lustre: Skipped 13 previous similar messages [ 1289.445977] Lustre: Modifying parameter general.timeout=55 in log params [ 1289.448657] Lustre: Skipped 25 previous similar messages [ 1321.709144] Lustre: Modifying parameter general.timeout=105 in log params [ 1321.714815] Lustre: Skipped 49 previous similar messages [ 1342.555302] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:35:49 (1780464949) [ 1344.694438] Lustre: 57126:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1346.002192] Lustre: 57225:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1348.424117] Lustre: 57421:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1348.428553] Lustre: 57421:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1365.901285] Lustre: 58984:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1365.906015] Lustre: 58984:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1365.909717] LustreError: 58984:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1368.015501] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:36:15 (1780464975) [ 1376.743791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1376.754338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.766844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.770243] Lustre: Skipped 2 previous similar messages [ 1381.422901] Lustre: server umount lustre-MDT0000 complete [ 1383.404459] LustreError: 56196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.420898] LustreError: 56196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1383.689483] LustreError: 48735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464990 with bad export cookie 15514384794966530827 [ 1383.692446] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.696301] LustreError: 48735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1383.915904] Lustre: server umount lustre-MDT0001 complete [ 1397.306077] Lustre: server umount lustre-OST0000 complete [ 1414.755685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.961484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1414.997315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1415.014335] Lustre: Skipped 47 previous similar messages [ 1415.271983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.490898] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1424.154704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.231143] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1424.255076] Lustre: 61511:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1427.780186] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1431.802503] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1434.516022] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1439.193376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.314267] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1444.868706] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1448.947195] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1450.275724] Lustre: 63736:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1457.632709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1457.637285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.644749] Lustre: Skipped 2 previous similar messages [ 1457.648743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1457.652844] Lustre: Skipped 3 previous similar messages [ 1460.672479] Lustre: server umount lustre-OST0000 complete [ 1468.386363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.389133] Lustre: Skipped 5 previous similar messages [ 1469.409300] Lustre: server umount lustre-MDT0000 complete [ 1471.941199] LustreError: 63146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465079 with bad export cookie 15514384794966533508 [ 1471.948027] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.951290] LustreError: 63146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1472.202869] Lustre: server umount lustre-MDT0001 complete [ 1480.368900] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1482.475910] Key type lgssc unregistered [ 1482.718810] LNet: 64928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.725992] LNetError: 64928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.741377] LNet: Removed LNI 192.168.202.160@tcp [ 1483.462150] Key type .llcrypt unregistered [ 1483.463213] Key type ._llcrypt unregistered [ 1500.179900] Key type ._llcrypt registered [ 1500.181499] Key type .llcrypt registered [ 1500.241621] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1501.113335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.147658] alg: No test for adler32 (adler32-zlib) [ 1502.127674] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1502.301080] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1503.943168] Key type lgssc registered [ 1504.869322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.110318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.126901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.507860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.474705] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1520.732993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.149621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1523.961504] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1527.807830] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1530.552240] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1535.335077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.639185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.693242] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1543.179875] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1553.581651] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1559.124987] Lustre: server umount lustre-MDT0000 complete [ 1561.524932] LustreError: 68645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465168 with bad export cookie 15933114678176077691 [ 1561.529791] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.532934] LustreError: 68645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1561.553927] LustreError: 66294:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1561.569382] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.574986] LustreError: 66298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.575498] Lustre: Skipped 1 previous similar message [ 1561.588714] LustreError: 66298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1561.595025] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1563.833708] Lustre: server umount lustre-MDT0001 complete [ 1567.058925] Lustre: server umount lustre-OST0000 complete [ 1585.421709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.591105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1585.626139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1585.822418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.181234] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1592.650304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1592.707031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1592.724254] Lustre: 70823:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1592.983931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1595.545320] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1599.390255] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1601.969227] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1606.279231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.392262] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1606.626082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.669623] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1612.628501] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1613.590762] Lustre: 73060:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1614.755580] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1619.530473] Lustre: server umount lustre-OST0000 complete [ 1622.496892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.501311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1622.508204] Lustre: Skipped 1 previous similar message [ 1627.616160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.620502] Lustre: Skipped 2 previous similar messages [ 1631.758569] Lustre: server umount lustre-MDT0000 complete [ 1632.738057] LustreError: 70830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.744959] LustreError: 70830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1633.968530] LustreError: 70816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465241 with bad export cookie 15933114678176078482 [ 1633.974158] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.977099] LustreError: 70816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1644.597821] Lustre: server umount lustre-MDT0001 complete [ 1651.745915] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1653.642395] Key type lgssc unregistered [ 1653.797850] LNet: 74253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.801377] LNetError: 74253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.809647] LNet: Removed LNI 192.168.202.160@tcp [ 1654.291794] Key type .llcrypt unregistered [ 1654.295177] Key type ._llcrypt unregistered [ 1670.590110] Key type ._llcrypt registered [ 1670.591864] Key type .llcrypt registered [ 1670.657676] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1671.417753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.443091] alg: No test for adler32 (adler32-zlib) [ 1672.471097] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1672.703918] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1674.335227] Key type lgssc registered [ 1675.302318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1680.546771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1680.561425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.916575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.818762] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1689.797887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.135153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1692.443316] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1696.082400] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1698.425376] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1702.565252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1702.936324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.866885] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1710.022368] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1719.095809] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1720.093565] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1720.098361] Lustre: Skipped 1 previous similar message [ 1721.196605] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1721.200343] Lustre: Skipped 1 previous similar message [ 1723.647165] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1723.650437] Lustre: Skipped 3 previous similar messages [ 1728.121494] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1728.124677] Lustre: Skipped 7 previous similar messages [ 1736.605221] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1736.607664] Lustre: Skipped 15 previous similar messages [ 1753.008906] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1753.011937] Lustre: Skipped 31 previous similar messages [ 1774.396685] Lustre: 83363:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1776.019724] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:43:03 (1780465383) [ 1777.126967] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1778.213765] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:43:05 (1780465385) [ 1781.730295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.736523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.737962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.741348] Lustre: Skipped 1 previous similar message [ 1781.747980] Lustre: Skipped 1 previous similar message [ 1786.782526] Lustre: server umount lustre-OST0000 complete [ 1791.977380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.990370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.997670] Lustre: Skipped 1 previous similar message [ 1795.511906] Lustre: server umount lustre-MDT0000 complete [ 1797.088643] LustreError: 75626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.104594] LustreError: 75626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1798.042323] LustreError: 75606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465405 with bad export cookie 3588277924733743658 [ 1798.059543] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.331936] Lustre: server umount lustre-MDT0001 complete [ 1806.068466] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1807.886949] Key type lgssc unregistered [ 1808.082676] LNet: 84594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.088166] LNetError: 84594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.102199] LNet: Removed LNI 192.168.202.160@tcp [ 1808.629559] Key type .llcrypt unregistered [ 1808.631549] Key type ._llcrypt unregistered [ 1810.840801] Key type ._llcrypt registered [ 1810.842289] Key type .llcrypt registered [ 1810.902874] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1813.081829] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules [ 1813.807876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1813.821069] alg: No test for adler32 (adler32-zlib) [ 1814.712242] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1814.836515] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1816.447892] Key type lgssc registered [ 1817.156484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1817.849910] LustreError: 85714:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1829.212180] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1834.964302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1838.908803] LustreError: 85714:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1838.952336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.485655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.885627] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1847.245361] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:44:14 (1780465454) [ 1849.826648] LustreError: 86721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.833759] LustreError: 86721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1849.924717] LustreError: 86753:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1849.934384] LustreError: 86753:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1849.941524] LustreError: 86753:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 1850.185493] Lustre: server umount lustre-MDT0000 complete [ 1856.562616] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1857.902330] Key type lgssc unregistered [ 1858.061597] LNet: 88209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.065811] LNetError: 88209:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.076625] LNet: Removed LNI 192.168.202.160@tcp [ 1858.485976] Key type .llcrypt unregistered [ 1858.487747] Key type ._llcrypt unregistered [ 1868.988918] Key type ._llcrypt registered [ 1868.990276] Key type .llcrypt registered [ 1869.049202] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1869.725392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.740535] alg: No test for adler32 (adler32-zlib) [ 1870.692910] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1870.855103] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1872.495155] Key type lgssc registered [ 1873.063027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.644743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.657968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.024561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.549905] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1886.573027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.958093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.443407] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1892.635901] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1894.595187] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1898.403319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.762816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.063714] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1904.927765] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1939.630412] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:45:46 (1780465546) [ 1940.585935] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1941.637327] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:45:48 (1780465548) [ 1947.104757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.109780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1947.112073] Lustre: Skipped 2 previous similar messages [ 1947.118481] Lustre: Skipped 2 previous similar messages [ 1949.903852] Lustre: server umount lustre-MDT0000 complete [ 1952.117294] LustreError: 89368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465559 with bad export cookie 15885355030204275383 [ 1952.126700] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.225279] LustreError: 89384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.233316] LustreError: 89384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1952.239759] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.338309] Lustre: server umount lustre-MDT0001 complete [ 1960.872659] Lustre: server umount lustre-OST0000 complete [ 1968.184738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.481152] LustreError: 93322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.493995] LustreError: 93322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1968.576808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.449321] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1973.729706] LustreError: 93323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.175728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.592290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1980.266755] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1983.246211] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1985.273952] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1992.996714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.186908] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1993.192527] LustreError: 93310:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1993.222506] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1993.233598] LustreError: 95177:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1993.237903] LustreError: 95177:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1993.246278] LustreError: 95177:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1993.288129] Lustre: server umount lustre-OST0000 complete [ 1993.290526] LustreError: 95177:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1998.515414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1998.644139] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1998.650447] LustreError: 93310:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1998.658149] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1998.661849] LustreError: 95642:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1998.666716] LustreError: 95642:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1998.669848] LustreError: 95642:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1998.715927] Lustre: server umount lustre-OST0000 complete [ 1998.720536] LustreError: 95642:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2004.809734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.917440] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2004.919799] Lustre: Found index 0 for lustre-OST0000, updating log [ 2004.924075] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2004.973861] Lustre: lustre-OST0000: new disk, initializing [ 2004.976855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2004.981982] Lustre: 96175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2005.147772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.797654] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2010.828419] Lustre: Failing over lustre-OST0000 [ 2010.923968] Lustre: server umount lustre-OST0000 complete [ 2013.154916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2013.162854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.169271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.272824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.273016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.176576] Lustre: server umount lustre-MDT0000 complete [ 2021.338757] LustreError: 93303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465628 with bad export cookie 15885355030204278113 [ 2021.345861] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.346674] LustreError: 93303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2021.544914] Lustre: server umount lustre-MDT0001 complete [ 2027.088338] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:47:14 (1780465634) [ 2036.476646] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2042.165629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.396665] LustreError: 98481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.466490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.328488] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2047.458147] LustreError: 98482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.016607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.027275] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2055.852891] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2059.538368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.363240] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2068.468327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2068.542636] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2068.599285] Lustre: lustre-OST0001: new disk, initializing [ 2068.605318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2068.611836] Lustre: 100747:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2068.615563] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 2068.628011] Lustre: Skipped 1 previous similar message [ 2072.053285] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2074.090636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2074.096515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2074.122735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2076.603634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2084.197276] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2088.417748] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2088.426257] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.440881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2098.902126] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 2098.912701] Lustre: Skipped 8 previous similar messages [ 2101.215134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2101.300076] Lustre: server umount lustre-MDT0001 complete [ 2104.025957] LustreError: 98477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.432337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.497124] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2107.499820] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2107.520705] Lustre: 98470:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.952995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2107.959354] Lustre: Skipped 3 previous similar messages [ 2108.988988] LustreError: 102315:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2108.997032] LustreError: 102315:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2109.006755] LustreError: 98474:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2109.087165] LustreError: 102327:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.160@tcp: cfg command failed: rc = -17 [ 2109.091197] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2109.091197] [ 2110.810347] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2113.020741] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2116.506375] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:48:43 (1780465723) [ 2125.569165] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2135.565130] Lustre: 104302:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2148.393353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2150.371657] Lustre: 105435:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2215.393720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.394468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2215.397689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.397698] Lustre: Skipped 3 previous similar messages [ 2215.398322] Lustre: Skipped 4 previous similar messages [ 2217.051497] Lustre: server umount lustre-MDT0000 complete [ 2219.100961] LustreError: 98463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465826 with bad export cookie 15885355030204279037 [ 2219.101575] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2219.110992] LustreError: 98463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2219.300226] Lustre: server umount lustre-MDT0001 complete [ 2231.527133] Lustre: server umount lustre-OST0000 complete [ 2243.005444] Lustre: server umount lustre-OST0001 complete [ 2245.750376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2250.235973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2254.687606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2260.730913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2271.573255] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2276.855470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.868893] Lustre: lustre-MDT0000: reset Object Index mappings [ 2277.117091] LustreError: 109366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.123782] LustreError: 109366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2277.167169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.303667] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2283.950907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.976223] Lustre: lustre-MDT0001: reset Object Index mappings [ 2286.706472] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2288.293097] Lustre: 110481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2292.400461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.875631] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2296.783130] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2302.003180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2303.405964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2303.411918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2303.478509] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2305.446326] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2310.519878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2312.677674] Lustre: 112321:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2342.197443] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:52:29 (1780465949) [ 2371.561642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.569842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.578127] Lustre: Skipped 1 previous similar message [ 2371.581090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.585485] Lustre: Skipped 3 previous similar messages [ 2385.375138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2385.511205] Lustre: server umount lustre-MDT0000 complete [ 2387.883071] LustreError: 109348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465995 with bad export cookie 15885355030204325867 [ 2387.885983] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.894913] LustreError: 109348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2390.501520] LustreError: 109362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.511321] LustreError: 109362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2402.271565] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2402.450735] Lustre: server umount lustre-MDT0001 complete [ 2419.167159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2419.245551] Lustre: server umount lustre-OST0000 complete [ 2427.669243] Lustre: server umount lustre-OST0001 complete [ 2431.763732] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2437.140163] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2457.014877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.151298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2457.170834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2457.208962] Lustre: lustre-MDT0000: new disk, initializing [ 2457.268534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.271147] Lustre: Skipped 3 previous similar messages [ 2457.282910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2459.351338] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2463.705307] Lustre: server umount lustre-MDT0000 complete [ 2468.086326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.296709] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2468.304408] Lustre: 116533:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2468.312382] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2470.578509] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2473.255290] Lustre: server umount lustre-MDT0000 complete [ 2477.517674] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:54:44 (1780466084) [ 2478.267441] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2479.194413] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:54:46 (1780466086) [ 2487.969710] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2493.296448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.587600] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2499.979843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.025475] Lustre: 118550:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2500.061327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2500.064332] Lustre: Skipped 1 previous similar message [ 2500.111438] Lustre: lustre-MDT0001: new disk, initializing [ 2500.195489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2500.203713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2501.981354] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2504.950475] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2508.330060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2508.480967] Lustre: lustre-OST0000: new disk, initializing [ 2508.483658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2508.484572] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2508.487712] Lustre: 120095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2508.490590] Lustre: Skipped 20 previous similar messages [ 2510.422821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2510.430438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2510.471161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2512.085886] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2518.621473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.728645] Lustre: lustre-OST0001: new disk, initializing [ 2518.731405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2518.735384] Lustre: 120945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2520.698178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2520.711337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2520.765763] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2521.678953] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2527.060343] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2528.786427] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2694.382722] Lustre: 127306:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2694.388143] Lustre: 127306:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2694.390811] Lustre: 127306:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2694.393866] Lustre: 127306:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2694.396805] Lustre: 127306:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2694.398878] Lustre: 127306:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2817.913245] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:00:25 (1780466425) [ 2832.865312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2832.869877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.875647] Lustre: Skipped 5 previous similar messages [ 2832.877914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.801161] Lustre: server umount lustre-MDT0000 complete [ 2837.364190] LustreError: 118541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780466444 with bad export cookie 15885355030204387040 [ 2837.368373] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2837.373710] LustreError: 118541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2838.497408] LustreError: 118555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.505127] LustreError: 118555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2843.604122] Lustre: server umount lustre-MDT0001 complete [ 2851.150271] Lustre: server umount lustre-OST0000 complete [ 2867.167271] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2867.279960] Lustre: server umount lustre-OST0001 complete [ 2870.089324] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2874.178069] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2888.299510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.396785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2888.411631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2888.449303] Lustre: lustre-MDT0000: new disk, initializing [ 2888.479785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.482610] Lustre: Skipped 5 previous similar messages [ 2888.489867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2890.049211] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2895.376889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.439992] Lustre: 130310:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2895.600419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2897.498121] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2901.195417] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2902.995125] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2905.849834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2905.976335] Lustre: 132048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2907.915488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2907.923439] Lustre: Skipped 1 previous similar message [ 2907.926113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2907.979677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2908.235806] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2911.573601] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2915.521204] Lustre: lustre-MDD0000: changelog on [ 2916.577036] Lustre: lustre-MDD0001: changelog on [ 3354.944294] Lustre: 132056:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3381.488095] Lustre: lustre-MDD0001: changelog off [ 3382.635357] Lustre: lustre-MDD0000: changelog off [ 3384.289105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.290385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.292710] Lustre: Skipped 6 previous similar messages [ 3384.296586] Lustre: Skipped 9 previous similar messages [ 3390.228058] Lustre: server umount lustre-OST0000 complete [ 3392.479722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3397.628916] Lustre: server umount lustre-MDT0000 complete [ 3398.807217] LustreError: 131096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467006 with bad export cookie 15885355030205190612 [ 3398.810036] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.811563] LustreError: 131096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3398.924551] Lustre: server umount lustre-MDT0001 complete [ 3402.296863] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3403.205323] Key type lgssc unregistered [ 3403.325362] LNet: 140729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3403.327556] LNetError: 140729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3403.337306] LNet: Removed LNI 192.168.202.160@tcp [ 3403.598115] Key type .llcrypt unregistered [ 3403.599099] Key type ._llcrypt unregistered [ 3409.459897] Key type ._llcrypt registered [ 3409.460789] Key type .llcrypt registered [ 3409.495290] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3415.369537] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3415.761674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.771720] alg: No test for adler32 (adler32-zlib) [ 3416.623884] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3416.703816] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3418.287173] Key type lgssc registered [ 3418.637395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3428.944591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3428.949455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.030434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3430.041407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3430.074917] Lustre: lustre-MDT0000: new disk, initializing [ 3430.097842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.105566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3431.351846] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3435.308332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.343640] Lustre: 143871:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3435.363974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3435.366074] Lustre: Skipped 1 previous similar message [ 3435.399012] Lustre: lustre-MDT0001: new disk, initializing [ 3435.430201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3435.443172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3435.447681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3436.697265] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3439.640576] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3440.765884] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3442.858305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.952732] Lustre: lustre-OST0000: new disk, initializing [ 3442.954507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3442.956726] Lustre: 145610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3442.984561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.469453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3444.472965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3444.503545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3444.696141] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3447.608119] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3449.714520] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_module obdecho/obdecho [ 3450.292585] LustreError: 146483:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3450.294798] LustreError: 146483:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3450.832496] LustreError: 146579:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3454.944959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.946330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.948255] Lustre: Skipped 2 previous similar messages [ 3454.950609] Lustre: Skipped 2 previous similar messages [ 3459.860883] Lustre: server umount lustre-MDT0000 complete [ 3461.025163] LustreError: 144647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467068 with bad export cookie 10035708368483181847 [ 3461.028259] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.028463] LustreError: 144647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3461.129363] Lustre: server umount lustre-MDT0001 complete [ 3472.604860] Lustre: server umount lustre-OST0000 complete [ 3479.411332] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3482.619585] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3494.284170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.378199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3494.390351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3494.425697] Lustre: lustre-MDT0000: new disk, initializing [ 3494.450215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.457761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3495.713571] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3500.000198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.045225] Lustre: 150253:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.076223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3500.078029] Lustre: Skipped 1 previous similar message [ 3500.117848] Lustre: lustre-MDT0001: new disk, initializing [ 3500.150279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3500.162951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3500.165691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3501.406368] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3504.529674] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3505.720496] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3507.883547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3507.986360] Lustre: 151996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3508.790373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3508.794811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3508.830599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3509.912670] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3512.938257] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3514.228682] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3514.300920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3515.614125] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3515.688417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3518.353480] Lustre: server umount lustre-OST0000 complete [ 3520.991784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3520.995271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.000174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.064775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.064931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.184243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.186648] Lustre: Skipped 2 previous similar messages [ 3529.212511] Lustre: server umount lustre-MDT0000 complete [ 3530.387403] LustreError: 150245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467137 with bad export cookie 10035708368483182799 [ 3530.390438] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.392092] LustreError: 150245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3539.982849] Lustre: server umount lustre-MDT0001 complete [ 3541.133794] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:12:28 (1780467148) [ 3541.665839] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3542.243925] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:12:29 (1780467149) [ 3544.665787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.812293] LustreError: 154258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.818573] LustreError: 154258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3544.838852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.841749] Lustre: Skipped 1 previous similar message [ 3546.161416] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3549.114766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.474838] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3552.233519] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3553.462089] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3555.828018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3557.932733] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3559.855603] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3574.752140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3574.754218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.757990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3578.651151] Lustre: server umount lustre-MDT0000 complete [ 3579.955326] LustreError: 154240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467187 with bad export cookie 10035708368483183667 [ 3579.958484] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3579.960090] LustreError: 154240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3580.628057] LustreError: 155765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.897258] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.903105] Lustre: Skipped 2 previous similar messages [ 3584.927817] LustreError: 154254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.928140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3584.934046] LustreError: 154254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3584.936555] Lustre: Skipped 6 previous similar messages [ 3586.159877] Lustre: server umount lustre-MDT0001 complete [ 3588.353389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3592.621871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.706130] LustreError: 157784:0:(llog.c:1646:llog_backup()) MGC192.168.202.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3592.709249] Lustre: 157784:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3610.591992] LustreError: 157789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.597671] LustreError: 157789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3615.711935] LustreError: 157789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.717684] LustreError: 157789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3624.931780] LustreError: 157789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.939040] LustreError: 157789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3641.044389] LustreError: 157790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.050747] LustreError: 157790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3653.727225] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3653.729252] Lustre: 157784:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3675.616094] LustreError: 157790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.622187] LustreError: 157790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3694.566833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3694.569205] Lustre: Skipped 2 previous similar messages [ 3695.769276] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3698.336755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.683460] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3700.383089] LustreError: 158508:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3700.385743] LustreError: 158508:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000a1857f0b x1866954989202688/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1780467307 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3700.392108] LustreError: 158508:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3700.703698] LustreError: 142094:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa860529 x1866954989206912/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3702.179061] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3704.031803] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3704.592592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3705.820360] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3705.829301] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3707.379456] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 3713.953723] Lustre: server umount lustre-MDT0000 complete [ 3715.198220] LustreError: 155872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467322 with bad export cookie 10035708368483190107 [ 3715.202018] LustreError: 155872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3715.203324] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.305480] Lustre: server umount lustre-MDT0001 complete [ 3735.960462] Lustre: server umount lustre-OST0000 complete [ 3742.344393] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3745.244795] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3756.265376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.355375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3756.367898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3756.369617] Lustre: Skipped 1 previous similar message [ 3756.404386] Lustre: lustre-MDT0000: new disk, initializing [ 3756.405935] Lustre: Skipped 1 previous similar message [ 3756.431980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.434047] Lustre: Skipped 1 previous similar message [ 3756.441230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.735629] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3761.944617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.985633] Lustre: 162597:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3762.018645] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3762.021601] Lustre: Skipped 1 previous similar message [ 3762.100207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3763.402893] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3766.414175] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3767.541126] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3769.691067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.791124] Lustre: 164341:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3771.643884] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3774.711891] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3774.971118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3775.005795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3776.053828] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3776.129421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3777.484063] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3777.552751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3780.066564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.067396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.069164] LustreError: Skipped 1 previous similar message [ 3780.075401] Lustre: Skipped 1 previous similar message [ 3784.269225] Lustre: server umount lustre-OST0000 complete [ 3787.743772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3791.641339] Lustre: server umount lustre-MDT0000 complete [ 3792.892380] LustreError: 162590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467400 with bad export cookie 10035708368483206907 [ 3792.896229] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.898571] LustreError: 162590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3793.012117] Lustre: server umount lustre-MDT0001 complete [ 3795.981593] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:16:43 (1780467403) [ 3798.196179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.355953] LustreError: 166602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.361302] LustreError: 166602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 3799.672067] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3802.444049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.941741] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3805.713668] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3806.827033] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3809.002791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3810.903577] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3812.590765] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.380631] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:17:20 (1780467440) [ 3833.824852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.825550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3833.826159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3833.826163] Lustre: Skipped 5 previous similar messages [ 3833.829216] Lustre: Skipped 3 previous similar messages [ 3839.436430] Lustre: server umount lustre-OST0000 complete [ 3843.551746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3846.742732] Lustre: server umount lustre-MDT0000 complete [ 3847.953210] LustreError: 166582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467455 with bad export cookie 10035708368483207789 [ 3847.956500] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3847.958498] LustreError: 166582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.076502] Lustre: server umount lustre-MDT0001 complete [ 3851.546944] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3852.613202] Key type lgssc unregistered [ 3852.737422] LNet: 169990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3852.740560] LNetError: 169990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3852.748354] LNet: Removed LNI 192.168.202.160@tcp [ 3853.072265] Key type .llcrypt unregistered [ 3853.074108] Key type ._llcrypt unregistered [ 3860.802668] Key type ._llcrypt registered [ 3860.803804] Key type .llcrypt registered [ 3860.848766] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3861.182968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3861.204967] alg: No test for adler32 (adler32-zlib) [ 3862.051864] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3862.135288] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3863.711183] Key type lgssc registered [ 3864.072611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3866.525391] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3866.532105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3881.951317] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3887.071143] LustreError: 171192:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.160@tcp: failed processing log, type 4: rc = -110 [ 3892.191381] LustreError: 171192:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3892.196247] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3892.198283] LustreError: MGC192.168.202.160@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3892.202408] LustreError: 171192:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3892.205690] LustreError: 171192:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 3892.208413] LustreError: 171192:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 3892.229924] Lustre: server umount lustre-OST0000 complete [ 3892.232214] LustreError: 171192:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3895.071452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.221920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.496529] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3900.308354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.428713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3901.717432] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3903.482404] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3904.657042] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3906.880184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.958169] LustreError: 173278:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3906.964950] Lustre: 173278:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.160@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3907.011490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.933078] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3910.768104] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3915.233281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.234677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.238621] Lustre: Skipped 2 previous similar messages [ 3915.240943] Lustre: Skipped 1 previous similar message [ 3917.650303] Lustre: server umount lustre-MDT0000 complete [ 3918.876217] LustreError: 171673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467526 with bad export cookie 11580778014092155764 [ 3918.878925] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3918.880730] LustreError: 171673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3918.992942] Lustre: server umount lustre-MDT0001 complete [ 3930.569992] Lustre: server umount lustre-OST0000 complete [ 3935.701434] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3941.301578] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3956.695771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.783996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3956.797722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3956.836795] Lustre: lustre-MDT0000: new disk, initializing [ 3956.865990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3956.873977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3958.428869] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3962.920807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.967441] Lustre: 176648:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3962.988993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3962.991052] Lustre: Skipped 1 previous similar message [ 3963.026360] Lustre: lustre-MDT0001: new disk, initializing [ 3963.061447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3963.074479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3963.077843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3964.472745] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3967.566858] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3968.748853] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3971.030515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3971.137690] Lustre: lustre-OST0000: new disk, initializing [ 3971.140230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3971.144223] Lustre: 178387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3971.174916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.002471] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3976.151430] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3977.215892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3977.219503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3977.266686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3977.692729] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3977.776883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3979.303938] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3979.388223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3982.305426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3982.308188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.314269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3982.317076] Lustre: Skipped 2 previous similar messages [ 3986.191410] Lustre: server umount lustre-OST0000 complete [ 3987.424224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.424697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3987.428737] Lustre: Skipped 2 previous similar messages [ 3987.432014] Lustre: Skipped 1 previous similar message [ 3992.543961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3992.546214] Lustre: Skipped 1 previous similar message [ 3993.569908] Lustre: server umount lustre-MDT0000 complete [ 3994.827351] LustreError: 176638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467602 with bad export cookie 11580778014092156464 [ 3994.830080] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3994.831759] LustreError: 176638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3994.952910] Lustre: server umount lustre-MDT0001 complete [ 3997.865483] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:20:05 (1780467605) [ 4003.819671] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4004.810605] Key type lgssc unregistered [ 4004.935411] LNet: 181165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4004.938503] LNetError: 181165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4004.945432] LNet: Removed LNI 192.168.202.160@tcp [ 4005.254311] Key type .llcrypt unregistered [ 4005.255543] Key type ._llcrypt unregistered [ 4012.112846] Key type ._llcrypt registered [ 4012.113745] Key type .llcrypt registered [ 4012.152509] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4012.432799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4012.452608] alg: No test for adler32 (adler32-zlib) [ 4013.298857] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4013.384460] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4014.967163] Key type lgssc registered [ 4015.313332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4017.888171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4017.895129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.049691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.367150] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4023.300582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.427544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4024.853480] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4026.969283] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4028.440411] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4031.408771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4031.482659] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4031.484051] LustreError: 183978:0:(llog.c:1687:llog_backup()) MGC192.168.202.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4031.487822] Lustre: 183978:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.160@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4031.524129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.496120] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4035.387541] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4038.627710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.629461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.632563] Lustre: Skipped 2 previous similar messages [ 4038.634773] Lustre: Skipped 1 previous similar message [ 4043.750520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.752493] Lustre: Skipped 2 previous similar messages [ 4048.864817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4048.866528] Lustre: Skipped 2 previous similar messages [ 4050.399085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4050.453227] Lustre: server umount lustre-MDT0000 complete [ 4051.576894] LustreError: 182323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467658 with bad export cookie 8561712199670488828 [ 4051.579617] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4051.580716] LustreError: 182323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4052.959251] LustreError: 183055:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b4fb6f7 x1866955615129728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4053.016150] Lustre: server umount lustre-MDT0001 complete [ 4064.598698] Lustre: server umount lustre-OST0000 complete [ 4067.873631] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4071.507619] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4084.180203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.264442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4084.275348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4084.310853] Lustre: lustre-MDT0000: new disk, initializing [ 4084.337387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.344150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4085.674063] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4090.077329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.116918] Lustre: 187352:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4090.134276] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4090.136243] Lustre: Skipped 1 previous similar message [ 4090.138389] LustreError: 188115:0:(llog.c:1687:llog_backup()) MGC192.168.202.160@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4090.142297] LustreError: 188115:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4090.144691] Lustre: 188115:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4090.154268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4090.156049] Lustre: Skipped 1 previous similar message [ 4090.192482] Lustre: lustre-MDT0001: new disk, initializing [ 4090.229960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4090.244917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4090.249396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4091.555030] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4094.739168] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4096.033328] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4098.403809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4098.496151] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4098.497949] Lustre: Skipped 1 previous similar message [ 4098.499590] LustreError: 189069:0:(llog.c:1687:llog_backup()) MGC192.168.202.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4098.502795] LustreError: 189069:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4098.504812] Lustre: 189069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4098.507793] Lustre: 189069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4098.516477] Lustre: lustre-OST0000: new disk, initializing [ 4098.517792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4098.519726] Lustre: 189093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4098.547144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.321367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4099.324376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4099.357844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4100.429646] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4103.601968] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4105.067635] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.153102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4106.536220] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4106.613131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4109.331852] Lustre: server umount lustre-OST0000 complete [ 4110.815717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4110.817632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.821030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.912139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.912711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.061578] Lustre: server umount lustre-MDT0000 complete [ 4121.315359] LustreError: 188128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467728 with bad export cookie 8561712199670489521 [ 4121.319452] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4121.324289] LustreError: 188128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4130.832771] Lustre: server umount lustre-MDT0001 complete [ 4133.820201] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:22:21 (1780467741) [ 4139.098802] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4142.246811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.375849] LustreError: 192051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.382677] LustreError: 192051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4142.390416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.589063] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4146.339262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.774236] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4148.725902] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4150.793973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4152.735416] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4155.387022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4155.443203] Lustre: lustre-OST0001: new disk, initializing [ 4155.445121] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4155.448075] Lustre: 194305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4157.247334] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4157.950183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4157.952684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4157.982800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4162.591264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4168.278089] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4168.866027] Lustre: DEBUG MARKER: ADD OST3 [ 4170.439540] loop: module loaded [ 4170.454993] loop0: detected capacity change from 0 to 204800000 [ 4170.527626] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4170.530994] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4170.535588] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4172.378465] loop0: detected capacity change from 0 to 204800000 [ 4174.142236] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4174.204118] Lustre: lustre-OST0003: new disk, initializing [ 4174.205383] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4174.207769] Lustre: 196018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4175.505134] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4175.513892] Lustre: DEBUG MARKER: STOP OST3 [ 4175.539634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4175.543212] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4175.554413] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4176.164084] Lustre: Failing over lustre-OST0003 [ 4176.213740] Lustre: server umount lustre-OST0003 complete [ 4176.215268] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4177.567105] LustreError: 196248:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4180.571605] Lustre: DEBUG MARKER: START OST3 again [ 4180.698113] LustreError: 193502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.964845] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.966066] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.771553] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.855807] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4182.858880] Lustre: Skipped 4 previous similar messages [ 4182.864104] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4184.472032] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4184.691345] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4184.809660] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4184.809741] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4197.800287] Lustre: Failing over lustre-OST0003 [ 4197.842930] Lustre: server umount lustre-OST0003 complete [ 4198.374219] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.375164] LustreError: 198063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.377353] Lustre: Skipped 2 previous similar messages [ 4198.382558] LustreError: 198063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4203.492371] LustreError: 193501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.499480] LustreError: 193501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4208.612830] LustreError: 193501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.618099] LustreError: 193501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4213.216935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4213.219474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.222708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.224427] Lustre: Skipped 3 previous similar messages [ 4216.360385] Lustre: server umount lustre-MDT0000 complete [ 4217.693518] LustreError: 192032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467824 with bad export cookie 8561712199670490389 [ 4217.698759] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4218.848110] LustreError: 192048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.854859] LustreError: 192048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4223.858344] Lustre: server umount lustre-MDT0001 complete [ 4225.313793] Lustre: server umount lustre-OST0000 complete [ 4226.617709] Lustre: server umount lustre-OST0001 complete [ 4228.805735] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4231.700165] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4243.284763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4243.366263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4243.377967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4243.417041] Lustre: lustre-MDT0000: new disk, initializing [ 4243.452522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4244.760682] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4248.879697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.923417] Lustre: 201361:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4249.013237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4249.015328] Lustre: Skipped 1 previous similar message [ 4249.027762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4250.289439] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4253.213687] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4254.369704] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4256.632969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4256.730158] Lustre: 203100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4258.294943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4258.326916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4258.460463] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4261.341534] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4262.528921] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4262.588340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4263.825868] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4263.896366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4268.514182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4268.514857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.516368] LustreError: Skipped 1 previous similar message [ 4268.522483] Lustre: Skipped 5 previous similar messages [ 4268.524607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4268.528009] Lustre: Skipped 8 previous similar messages [ 4270.611474] Lustre: server umount lustre-OST0000 complete [ 4277.909251] Lustre: server umount lustre-MDT0000 complete [ 4278.752240] LustreError: 201366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.756854] LustreError: 201366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4279.078318] LustreError: 201354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467886 with bad export cookie 8561712199670492643 [ 4279.081369] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4279.082016] LustreError: 201354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4279.196992] Lustre: server umount lustre-MDT0001 complete [ 4280.325248] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:24:48 (1780467888) [ 4285.316557] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4288.259453] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4299.280402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.362416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4299.373226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4299.374816] Lustre: Skipped 3 previous similar messages [ 4299.407668] Lustre: lustre-MDT0000: new disk, initializing [ 4299.409142] Lustre: Skipped 2 previous similar messages [ 4299.435185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4299.437167] Lustre: Skipped 2 previous similar messages [ 4300.673828] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4304.798445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.840170] Lustre: 207520:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4304.941848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4306.219564] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4309.135489] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4310.245415] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4312.426930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.520528] Lustre: 209260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4314.085742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4314.295235] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4317.212334] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4318.400099] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4318.469194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4319.656277] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4319.728513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4324.320856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4324.321817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.322976] LustreError: Skipped 1 previous similar message [ 4324.327235] Lustre: Skipped 2 previous similar messages [ 4324.327969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4324.332014] Lustre: Skipped 3 previous similar messages [ 4326.412152] Lustre: server umount lustre-OST0000 complete [ 4333.724399] Lustre: server umount lustre-MDT0000 complete [ 4334.560029] LustreError: 208304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.566429] LustreError: 208304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4335.027295] LustreError: 208297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467942 with bad export cookie 8561712199670493525 [ 4335.030919] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4335.032316] LustreError: 208297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4335.149710] Lustre: server umount lustre-MDT0001 complete [ 4337.937224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.350697] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4342.162216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.568806] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4345.351708] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4346.490444] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4348.630891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4350.467867] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4352.111818] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4377.568179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.985158] Lustre: server umount lustre-OST0000 complete [ 4387.414143] Lustre: server umount lustre-MDT0000 complete [ 4388.672124] LustreError: 211306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780467995 with bad export cookie 8561712199670494407 [ 4388.674830] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.675686] LustreError: 211306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4388.785489] Lustre: server umount lustre-MDT0001 complete [ 4486.659241] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4487.658840] Key type lgssc unregistered [ 4487.792345] LNet: 214652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4487.794622] LNetError: 214652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4487.807281] LNet: Removed LNI 192.168.202.160@tcp [ 4488.124113] Key type .llcrypt unregistered [ 4488.125367] Key type ._llcrypt unregistered [ 4496.379144] Key type ._llcrypt registered [ 4496.380134] Key type .llcrypt registered [ 4496.419385] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4502.079655] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4502.375549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4502.412422] alg: No test for adler32 (adler32-zlib) [ 4503.250915] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4503.325220] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4504.903210] Key type lgssc registered [ 4505.268273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4516.235821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4516.242019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.318545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4517.328419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4517.361650] Lustre: lustre-MDT0000: new disk, initializing [ 4517.383829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.389390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4518.587775] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4522.368135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.400687] Lustre: 217944:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4522.418149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4522.419521] Lustre: Skipped 1 previous similar message [ 4522.448087] Lustre: lustre-MDT0001: new disk, initializing [ 4522.475688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4522.485051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4522.487949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4523.703575] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4526.802606] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4527.919284] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4530.165045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4530.256770] Lustre: lustre-OST0000: new disk, initializing [ 4530.258229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4530.260414] Lustre: 219682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4530.284077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4532.120717] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4533.812571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4533.816756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4533.841153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4535.066430] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4536.383832] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4536.456117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4537.720401] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4537.786248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4538.848744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4538.849563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.851542] LustreError: Skipped 1 previous similar message [ 4538.852248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4538.855421] Lustre: Skipped 1 previous similar message [ 4540.426170] Lustre: server umount lustre-OST0000 complete [ 4542.943969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4542.947388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.952412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4542.954683] Lustre: Skipped 1 previous similar message [ 4543.968276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.968912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.665680] Lustre: server umount lustre-MDT0000 complete [ 4548.813322] LustreError: 217936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468156 with bad export cookie 15873497209990006105 [ 4548.816463] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.817920] LustreError: 217936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4548.925984] Lustre: server umount lustre-MDT0001 complete [ 4554.351118] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4557.499523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.630370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.806907] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4561.689141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.810576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4563.201311] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4564.207292] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4566.368029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4568.237424] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4570.901820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.956303] Lustre: lustre-OST0001: new disk, initializing [ 4570.958282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4570.961255] Lustre: 224701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4570.989341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4570.992210] Lustre: Skipped 1 previous similar message [ 4572.664823] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4573.131316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4573.136477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4573.171868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4578.140621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4580.822738] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4584.465545] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:29:52 (1780468192) [ 4587.488497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4587.490808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.496198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.499399] Lustre: Skipped 1 previous similar message [ 4591.254818] Lustre: server umount lustre-MDT0000 complete [ 4592.473273] LustreError: 223548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468199 with bad export cookie 15873497209990006973 [ 4592.476961] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.478709] LustreError: 223548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4592.596669] Lustre: server umount lustre-MDT0001 complete [ 4604.164786] Lustre: server umount lustre-OST0000 complete [ 4615.519289] Lustre: server umount lustre-OST0001 complete [ 4617.555523] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4620.842768] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4631.975555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.064414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4632.073141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4632.105156] Lustre: lustre-MDT0000: new disk, initializing [ 4632.124587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.129722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4633.295459] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4637.162262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.200975] Lustre: 228587:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4637.305481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4638.461546] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4641.351891] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4642.491572] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4644.771026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.875280] Lustre: lustre-OST0000: new disk, initializing [ 4644.876950] Lustre: Skipped 1 previous similar message [ 4644.878517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4644.881020] Lustre: Skipped 2 previous similar messages [ 4644.883572] Lustre: 230327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4646.457696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4646.461429] Lustre: Skipped 1 previous similar message [ 4646.464401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4646.503248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4646.872764] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4650.006559] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4651.377996] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4651.448794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4652.862645] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4652.942549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4655.698842] Lustre: server umount lustre-OST0000 complete [ 4657.632034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.634318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.640863] Lustre: Skipped 3 previous similar messages [ 4657.643718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4657.645505] Lustre: Skipped 4 previous similar messages [ 4666.849092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4666.850962] Lustre: Skipped 3 previous similar messages [ 4666.881209] Lustre: server umount lustre-MDT0000 complete [ 4668.183671] LustreError: 228579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468275 with bad export cookie 15873497209990009066 [ 4668.187479] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4668.189058] LustreError: 228579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4678.685155] Lustre: server umount lustre-MDT0001 complete [ 4681.740454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.925324] LustreError: 232395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.930468] LustreError: 232395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4681.962821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.966570] Lustre: Skipped 2 previous similar messages [ 4683.377230] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4686.193830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.569927] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4689.308209] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4690.484983] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4692.758391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.762707] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4696.563489] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4844.511817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.515064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.518773] Lustre: Skipped 1 previous similar message [ 4844.520444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.675402] Lustre: server umount lustre-OST0000 complete [ 4854.047031] Lustre: server umount lustre-MDT0000 complete [ 4855.316252] LustreError: 232375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468462 with bad export cookie 15873497209990009948 [ 4855.319533] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.320943] LustreError: 232375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4855.449325] Lustre: server umount lustre-MDT0001 complete [ 4961.791329] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4962.851724] Key type lgssc unregistered [ 4962.975400] LNet: 235651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4962.978285] LNetError: 235651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4962.988664] LNet: Removed LNI 192.168.202.160@tcp [ 4963.298156] Key type .llcrypt unregistered [ 4963.299295] Key type ._llcrypt unregistered [ 4971.621328] Key type ._llcrypt registered [ 4971.623499] Key type .llcrypt registered [ 4971.667109] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4977.859493] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4978.149664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.187452] alg: No test for adler32 (adler32-zlib) [ 4979.033202] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4979.113932] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4980.695132] Key type lgssc registered [ 4981.023750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4994.476333] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4997.813332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4997.818573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.912821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.927360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4998.961400] Lustre: lustre-MDT0000: new disk, initializing [ 4998.986274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.992541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5000.252431] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5004.107374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.146472] Lustre: 239641:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5004.167259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5004.168939] Lustre: Skipped 1 previous similar message [ 5004.202299] Lustre: lustre-MDT0001: new disk, initializing [ 5004.231789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.244488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5004.247515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5005.401483] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5007.541372] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5009.536197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5009.625451] Lustre: lustre-OST0000: new disk, initializing [ 5009.627637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5009.630532] Lustre: 241235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5009.658244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.333631] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5012.787895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5012.790460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5012.824496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5015.148036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.204858] Lustre: lustre-OST0001: new disk, initializing [ 5015.206451] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5015.208823] Lustre: 242089:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5015.239134] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.000677] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5019.139171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5019.142151] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5019.172964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5022.491815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5028.299030] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5034.466198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.467600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5034.470162] Lustre: Skipped 2 previous similar messages [ 5034.472580] Lustre: Skipped 1 previous similar message [ 5037.285930] Lustre: server umount lustre-MDT0000 complete [ 5038.518429] LustreError: 242840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468645 with bad export cookie 14145501202881826407 [ 5038.522957] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.523947] LustreError: 242840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5038.650118] Lustre: server umount lustre-MDT0001 complete [ 5050.188071] Lustre: server umount lustre-OST0000 complete [ 5061.735319] Lustre: server umount lustre-OST0001 complete [ 5066.345187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5069.536192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5072.379288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5075.535783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5081.747360] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5085.225317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.233058] Lustre: lustre-MDT0000: reset Object Index mappings [ 5085.418141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.722945] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5089.602729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.608829] Lustre: lustre-MDT0001: reset Object Index mappings [ 5091.179668] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5092.187261] Lustre: 248160:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5094.405783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5094.529377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5094.531943] Lustre: Skipped 1 previous similar message [ 5095.587595] LustreError: 248513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.591148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5095.593933] LustreError: 248513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5096.424680] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5099.329526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.459276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5101.311765] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5104.388937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5105.621095] Lustre: 249994:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5110.239983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5110.244873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.252700] Lustre: Skipped 1 previous similar message [ 5110.255887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5110.257924] Lustre: Skipped 2 previous similar messages [ 5112.929685] Lustre: server umount lustre-MDT0000 complete [ 5114.141208] LustreError: 247081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468721 with bad export cookie 14145501202881833183 [ 5114.144520] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.145854] LustreError: 247081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5114.265506] Lustre: server umount lustre-MDT0001 complete [ 5125.059697] Lustre: server umount lustre-OST0000 complete [ 5136.537638] Lustre: server umount lustre-OST0001 complete [ 5142.265318] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:39:10 (1780468750) [ 5147.095831] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 5149.957829] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5160.770919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.848845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5160.858055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5160.889471] Lustre: lustre-MDT0000: new disk, initializing [ 5160.909706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.911629] Lustre: Skipped 1 previous similar message [ 5160.916358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5162.028132] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5165.988241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.035396] Lustre: 254207:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5166.151760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5167.507407] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5170.429525] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5171.580699] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5173.671346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5173.780812] Lustre: lustre-OST0000: new disk, initializing [ 5173.782053] Lustre: Skipped 1 previous similar message [ 5173.783524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5173.784974] Lustre: Skipped 2 previous similar messages [ 5173.786708] Lustre: 255944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5175.036604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5175.039073] Lustre: Skipped 1 previous similar message [ 5175.040952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5175.080993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.596982] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5178.570324] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5179.868108] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5179.935614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5181.209837] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5181.284943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5185.507316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.508679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5185.509223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.509227] Lustre: Skipped 3 previous similar messages [ 5185.512696] Lustre: Skipped 4 previous similar messages [ 5185.515416] LustreError: Skipped 1 previous similar message [ 5190.626178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5190.629218] Lustre: Skipped 2 previous similar messages [ 5195.745683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.747533] Lustre: Skipped 1 previous similar message [ 5196.255142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5196.310964] Lustre: server umount lustre-OST0000 complete [ 5200.865341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.868645] Lustre: Skipped 1 previous similar message [ 5203.673342] Lustre: server umount lustre-MDT0000 complete [ 5204.871270] LustreError: 254983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468812 with bad export cookie 14145501202881835255 [ 5204.873937] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.874533] LustreError: 254983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5204.978726] Lustre: server umount lustre-MDT0001 complete [ 5210.779709] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5214.112385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.244517] LustreError: 258711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.260060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.262289] Lustre: Skipped 2 previous similar messages [ 5215.501433] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5218.202733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.561864] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5220.503915] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5222.663401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5224.552566] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5227.466594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5227.532883] Lustre: lustre-OST0001: new disk, initializing [ 5227.534936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5227.538558] Lustre: 260965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5228.667597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5228.670628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5228.701339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5229.463565] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5233.925561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5235.054204] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5236.658215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5236.666168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5238.752441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5238.755628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.761906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5238.764980] Lustre: Skipped 2 previous similar messages [ 5244.451787] Lustre: server umount lustre-MDT0000 complete [ 5245.716146] LustreError: 260964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468852 with bad export cookie 14145501202881836137 [ 5245.719630] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.720270] LustreError: 260964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5245.831127] Lustre: server umount lustre-MDT0001 complete [ 5257.739805] Lustre: server umount lustre-OST0000 complete [ 5269.283802] Lustre: server umount lustre-OST0001 complete [ 5274.591800] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5277.991724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.139970] LustreError: 263982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.484397] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5282.411359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5282.558493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5282.560713] Lustre: Skipped 4 previous similar messages [ 5283.882741] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5284.817532] Lustre: 265089:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5286.960134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5288.100413] LustreError: 265443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.119988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5289.104692] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5291.864372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5292.972702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5293.893939] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5296.921070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5298.159547] Lustre: 266925:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5299.946767] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5299.956423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5302.878786] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:41:50 (1780468910) [ 5318.627852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.629619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5318.632277] Lustre: Skipped 6 previous similar messages [ 5318.633218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5318.634442] Lustre: Skipped 11 previous similar messages [ 5320.537277] Lustre: server umount lustre-MDT0000 complete [ 5321.771211] LustreError: 264709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780468929 with bad export cookie 14145501202881842437 [ 5321.775809] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5321.776449] LustreError: 264709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5323.743817] LustreError: 263979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.751425] LustreError: 263979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5331.872360] LustreError: 263979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.877099] LustreError: 263979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5336.031101] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5336.114482] Lustre: server umount lustre-MDT0001 complete [ 5343.422662] Lustre: server umount lustre-OST0000 complete [ 5350.739496] Lustre: server umount lustre-OST0001 complete [ 5352.790730] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 5356.062485] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5370.605250] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5373.913739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5373.999587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5374.010528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5374.050416] Lustre: lustre-MDT0000: new disk, initializing [ 5374.080622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5375.379137] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5379.564560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.608650] Lustre: 271109:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5379.719483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5380.974267] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5385.499612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5385.605239] Lustre: 272707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5386.968591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5387.487078] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5391.623860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.319513] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5393.569682] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5397.871088] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5402.999283] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:43:30 (1780469010) [ 5405.663832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5405.666049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.670057] Lustre: Skipped 2 previous similar messages [ 5405.671785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5405.673456] Lustre: Skipped 6 previous similar messages [ 5409.828416] Lustre: server umount lustre-MDT0000 complete [ 5411.027236] LustreError: 271103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469018 with bad export cookie 14145501202881850319 [ 5411.030360] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.031176] LustreError: 271103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5411.139466] Lustre: server umount lustre-MDT0001 complete [ 5422.081621] Lustre: server umount lustre-OST0000 complete [ 5433.568573] Lustre: server umount lustre-OST0001 complete [ 5436.364013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.494718] LustreError: 275783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.520070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5436.521717] Lustre: Skipped 6 previous similar messages [ 5437.668134] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5440.245874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.527481] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5443.127792] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5444.170415] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5446.162448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5447.895979] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5449.572221] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5459.530254] loop1: detected capacity change from 0 to 204800000 [ 5459.616279] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5459.619625] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5459.624746] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5461.336647] loop1: detected capacity change from 0 to 204800000 [ 5463.170670] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.200510] LustreError: 275772:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs3603: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5463.204602] LustreError: 275772:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs3603 [ 5463.206232] LustreError: 275772:0:(mgs_handler.c:517:mgs_target_reg()) fs3603-MDT0001: Error checking target index: rc = -13 [ 5463.209790] LustreError: fs3603-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5463.214770] LustreError: 279158:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 5463.218770] LustreError: 279158:0:(tgt_mount.c:2084:server_put_super()) no obd fs3603-MDT0001 [ 5463.222205] LustreError: 279158:0:(tgt_mount.c:116:server_deregister_mount()) fs3603-MDT0001 not registered [ 5463.235493] Lustre: server umount fs3603-MDT0001 complete [ 5463.237159] LustreError: 279158:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5466.112084] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.141887] Lustre: Setting parameter fs3603-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3603-MDT0001 [ 5466.147403] Lustre: Skipped 2 previous similar messages [ 5466.162923] Lustre: srv-fs3603-MDT0001: No data found on store. Initialize space. [ 5466.164756] Lustre: Skipped 4 previous similar messages [ 5466.200889] Lustre: fs3603-MDT0001: new disk, initializing [ 5466.202224] Lustre: Skipped 3 previous similar messages [ 5466.236070] Lustre: srv-fs3603-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5466.238274] Lustre: 279622:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3603-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5466.240794] LustreError: 279622:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3603-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5467.509203] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5470.326129] Lustre: server umount fs3603-MDT0001 complete [ 5470.421025] LustreError: 277943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.426000] LustreError: 277943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5474.628271] loop1: detected capacity change from 0 to 204800000 [ 5476.512469] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.632464] Lustre: 281170:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3603-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5476.636054] LustreError: 281170:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3603-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5477.969208] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5479.560307] Lustre: server umount fs3603-MDT0001 complete [ 5485.789260] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 5485.791384] Lustre: Skipped 4 previous similar messages [ 5486.562762] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.564365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5486.568757] Lustre: Skipped 4 previous similar messages [ 5488.927825] Lustre: server umount lustre-MDT0000 complete [ 5491.834791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5491.874016] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.352781] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5495.949608] loop1: detected capacity change from 0 to 204800000 [ 5497.320092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5497.890315] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.013971] Lustre: 283581:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3603-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5498.018291] LustreError: 283581:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3603-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5499.369057] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5500.975524] Lustre: server umount fs3603-MDT0001 complete [ 5512.160174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5521.887139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5521.945716] Lustre: server umount lustre-OST0000 complete [ 5529.373719] Lustre: server umount lustre-MDT0000 complete [ 5530.677265] LustreError: 275763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469137 with bad export cookie 14145501202881852993 [ 5530.680556] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5530.680841] LustreError: 275763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5530.808197] Lustre: server umount lustre-MDT0001 complete [ 5534.333247] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5535.362021] Key type lgssc unregistered [ 5535.484495] LNet: 285887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5535.487071] LNetError: 285887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5535.494337] LNet: Removed LNI 192.168.202.160@tcp [ 5535.800143] Key type .llcrypt unregistered [ 5535.802044] Key type ._llcrypt unregistered [ 5543.313812] Key type ._llcrypt registered [ 5543.314925] Key type .llcrypt registered [ 5543.353278] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5543.767103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5543.776970] alg: No test for adler32 (adler32-zlib) [ 5544.631414] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5544.713864] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5546.295199] Key type lgssc registered [ 5546.647199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5549.189038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5549.196340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.360118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.668283] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5554.564282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.708162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5555.991414] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5557.745723] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5558.953234] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5561.179175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5561.297125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.167102] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5564.388308] LustreError: 288720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.394198] LustreError: 288720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5564.970182] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5569.508037] LustreError: 288720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.626400] LustreError: 288722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.673600] LustreError: 288721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.678427] LustreError: 288721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5581.796521] LustreError: 289286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.801842] LustreError: 289286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5586.515888] loop1: detected capacity change from 0 to 204800000 [ 5586.617922] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5586.622118] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5586.626390] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5588.332660] loop1: detected capacity change from 0 to 204800000 [ 5590.199716] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5590.253540] Lustre: fs27182-OST0000: new disk, initializing [ 5590.255139] Lustre: srv-fs27182-OST0000: No data found on store. Initialize space. [ 5590.257487] Lustre: 290534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5590.283255] Lustre: fs27182-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5590.741172] LustreError: 288720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.747602] LustreError: 288720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5592.155367] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5595.694369] Lustre: server umount fs27182-OST0000 complete [ 5599.604819] loop1: detected capacity change from 0 to 204800000 [ 5601.587620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.654641] Lustre: fs27182-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5603.446769] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5606.275165] loop2: detected capacity change from 0 to 204800000 [ 5606.362263] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5606.365940] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5606.370740] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5607.396307] LustreError: 288722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.401682] LustreError: 288722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5608.241161] loop2: detected capacity change from 0 to 204800000 [ 5610.223906] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.257359] LustreError: fs27182-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5610.261275] LustreError: fs27182-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5610.264094] LustreError: 293534:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 5610.266274] LustreError: 293534:0:(tgt_mount.c:2084:server_put_super()) no obd fs27182-OST0001 [ 5610.268039] LustreError: 293534:0:(tgt_mount.c:116:server_deregister_mount()) fs27182-OST0001 not registered [ 5610.278204] Lustre: server umount fs27182-OST0001 complete [ 5610.279669] LustreError: 293534:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5613.695504] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5613.739261] Lustre: fs27182-OST0001: new disk, initializing [ 5613.741471] Lustre: srv-fs27182-OST0001: No data found on store. Initialize space. [ 5613.744911] Lustre: 294087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5613.768435] Lustre: fs27182-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.706721] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5619.222272] Lustre: server umount fs27182-OST0000 complete [ 5622.257704] Lustre: server umount fs27182-OST0001 complete [ 5626.326126] loop1: detected capacity change from 0 to 204800000 [ 5628.218645] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5630.171946] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5632.388027] loop2: detected capacity change from 0 to 204800000 [ 5634.239249] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5634.304364] Lustre: fs27182-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5634.306163] Lustre: Skipped 1 previous similar message [ 5636.193495] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5639.265686] Lustre: server umount fs27182-OST0000 complete [ 5641.940535] LustreError: 289228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.947141] LustreError: 289228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 5642.432473] Lustre: server umount fs27182-OST0001 complete [ 5646.304152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.308185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.315116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5648.352657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5648.355183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.355880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5648.360246] Lustre: Skipped 1 previous similar message [ 5652.241990] Lustre: server umount lustre-OST0000 complete [ 5657.055957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5657.058122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.062098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5659.609967] Lustre: server umount lustre-MDT0000 complete [ 5660.844337] LustreError: 294086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469268 with bad export cookie 12121126104399425611 [ 5660.847395] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.847522] LustreError: 294086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5660.964403] Lustre: server umount lustre-MDT0001 complete [ 5664.508090] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5665.603664] Key type lgssc unregistered [ 5665.738352] LNet: 299929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5665.740653] LNetError: 299929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5665.748299] LNet: Removed LNI 192.168.202.160@tcp [ 5666.055143] Key type .llcrypt unregistered [ 5666.056830] Key type ._llcrypt unregistered [ 5676.285647] Key type ._llcrypt registered [ 5676.286619] Key type .llcrypt registered [ 5676.327184] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 5682.573295] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5682.954042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5682.992769] alg: No test for adler32 (adler32-zlib) [ 5683.851211] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5683.940967] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5685.527161] Key type lgssc registered [ 5685.905664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5697.628683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5697.637968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.730562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5698.743506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5698.782622] Lustre: lustre-MDT0000: new disk, initializing [ 5698.807872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.817158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5700.184737] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5704.316073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.357635] Lustre: 303725:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5704.383073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5704.385030] Lustre: Skipped 1 previous similar message [ 5704.420507] Lustre: lustre-MDT0001: new disk, initializing [ 5704.453232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.465342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5704.469842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5705.779780] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5708.908143] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5710.074187] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5712.246488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.343937] Lustre: lustre-OST0000: new disk, initializing [ 5712.346308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5712.348990] Lustre: 305465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5712.372512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.077530] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5716.213289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5716.215997] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5716.246450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5717.018159] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5717.614183] Lustre: Mounted lustre-client [ 5717.913122] Lustre: Unmounted lustre-client [ 5721.568158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.568253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.570647] LustreError: Skipped 1 previous similar message [ 5721.575106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5721.577558] Lustre: Skipped 1 previous similar message [ 5722.509798] Lustre: server umount lustre-OST0000 complete [ 5723.810975] Lustre: server umount lustre-MDT0000 complete [ 5725.031557] LustreError: 304504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469332 with bad export cookie 17450245247104966888 [ 5725.035294] LustreError: 304504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5725.035481] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.179748] Lustre: server umount lustre-MDT0001 complete [ 5734.951808] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5735.981406] Key type lgssc unregistered [ 5776.031300] LNet: 307492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5776.035261] LNetError: 307492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5846.693297] LNet: Removed LNI 192.168.202.160@tcp [ 5847.001362] Key type .llcrypt unregistered [ 5847.002350] Key type ._llcrypt unregistered [ 5852.641764] Key type ._llcrypt registered [ 5852.643011] Key type .llcrypt registered [ 5852.676677] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 5858.795822] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5859.148194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5859.172489] alg: No test for adler32 (adler32-zlib) [ 5860.028253] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5860.110185] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5861.687109] Key type lgssc registered [ 5862.030580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5871.806606] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:51:19 (1780469479) [ 5874.066415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5874.072779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.165532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5875.177187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5875.214450] Lustre: lustre-MDT0000: new disk, initializing [ 5875.240320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.246565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5876.552314] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5880.655937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.699125] Lustre: 310654:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5880.721268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5880.723147] Lustre: Skipped 1 previous similar message [ 5880.757931] Lustre: lustre-MDT0001: new disk, initializing [ 5880.789072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5880.800758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5880.803627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5882.069820] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5885.181682] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5886.397445] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5888.566306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5888.677983] Lustre: lustre-OST0000: new disk, initializing [ 5888.679779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5888.683167] Lustre: 312395:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5888.709588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.575275] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5893.711385] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5895.672523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5895.677182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5895.705981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5905.888884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.889786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.891541] LustreError: Skipped 1 previous similar message [ 5905.895950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.898770] Lustre: Skipped 1 previous similar message [ 5908.813543] Lustre: server umount lustre-OST0000 complete [ 5911.009063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.009788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.013527] Lustre: Skipped 2 previous similar messages [ 5911.016783] Lustre: Skipped 1 previous similar message [ 5916.128915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.131518] Lustre: Skipped 1 previous similar message [ 5916.259684] Lustre: server umount lustre-MDT0000 complete [ 5917.540083] LustreError: 310649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469524 with bad export cookie 14950315628375393302 [ 5917.543434] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.544092] LustreError: 310649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5917.660874] Lustre: server umount lustre-MDT0001 complete [ 5921.126486] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5922.071492] Key type lgssc unregistered [ 5922.184401] LNet: 314026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5922.186327] LNetError: 314026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5922.194299] LNet: Removed LNI 192.168.202.160@tcp [ 5922.470431] Key type .llcrypt unregistered [ 5922.472041] Key type ._llcrypt unregistered [ 5931.379222] Key type ._llcrypt registered [ 5931.380073] Key type .llcrypt registered [ 5931.412228] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5931.705479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5931.755239] alg: No test for adler32 (adler32-zlib) [ 5932.592739] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5932.668988] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5934.247131] Key type lgssc registered [ 5934.568360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5936.818193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5936.822997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.965555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.235422] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5941.781494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.885121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.004167] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5944.557184] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5945.611659] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5947.644367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5947.743377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.493243] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5951.079422] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5954.529351] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5955.006884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5957.089521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.091588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.092701] Lustre: Skipped 2 previous similar messages [ 5957.094405] Lustre: Skipped 1 previous similar message [ 5958.295252] Lustre: server umount lustre-MDT0000 complete [ 5959.556145] LustreError: 315385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469566 with bad export cookie 15248088075726857187 [ 5959.560301] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5959.561180] LustreError: 315385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5962.207878] LustreError: 317553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.208239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.212813] LustreError: 317553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5962.217735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5962.221148] Lustre: Skipped 2 previous similar messages [ 5964.255851] LustreError: 317553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.256128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5964.260935] LustreError: 317553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5964.264155] Lustre: Skipped 1 previous similar message [ 5965.737665] Lustre: server umount lustre-MDT0001 complete [ 5967.577725] Lustre: lustre-OST0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 5972.691765] Lustre: lustre-OST0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 5973.081131] Lustre: server umount lustre-OST0000 complete [ 5975.981330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.403241] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5978.393996] Lustre: 319129:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.160@tcp [ 5979.145962] Lustre: server umount MGS complete [ 5982.054380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.211234] LustreError: 319698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.236635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.538077] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5986.336779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.451063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5987.760942] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5990.686556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.745887] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5994.621489] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5999.282916] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6000.416363] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6000.890936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6006.241798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6006.242557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.244654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.247900] Lustre: Skipped 1 previous similar message [ 6009.422084] Lustre: server umount lustre-OST0000 complete [ 6011.360890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6011.366804] Lustre: Skipped 1 previous similar message [ 6016.795909] Lustre: server umount lustre-MDT0000 complete [ 6018.032135] LustreError: 321849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469625 with bad export cookie 15248088075726858013 [ 6018.035566] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.036017] LustreError: 321849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6018.147136] Lustre: server umount lustre-MDT0001 complete [ 6021.630749] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6022.636819] Key type lgssc unregistered [ 6022.766329] LNet: 322872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6022.769876] LNetError: 322872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6022.778279] LNet: Removed LNI 192.168.202.160@tcp [ 6023.064337] Key type .llcrypt unregistered [ 6023.065365] Key type ._llcrypt unregistered [ 6029.079747] Key type ._llcrypt registered [ 6029.081451] Key type .llcrypt registered [ 6029.122516] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6035.387369] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6035.726396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6035.780780] alg: No test for adler32 (adler32-zlib) [ 6036.634972] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6036.718442] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 6038.303159] Key type lgssc registered [ 6038.684061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6050.337635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6050.362878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.450199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6051.463327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6051.506381] Lustre: lustre-MDT0000: new disk, initializing [ 6051.530246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.536442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6052.821274] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6056.904349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.951452] Lustre: 325971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6056.976820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6056.978788] Lustre: Skipped 1 previous similar message [ 6057.033504] Lustre: lustre-MDT0001: new disk, initializing [ 6057.068921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6057.081072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6057.084025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6058.344389] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6061.364652] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6062.538757] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6064.739558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6064.844422] Lustre: lustre-OST0000: new disk, initializing [ 6064.846151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6064.849238] Lustre: 327715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6064.881674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.698392] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6069.755193] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6071.112265] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6071.286564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6071.289886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6071.320822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6072.208590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6073.551705] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6073.629526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6076.386157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.390257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.390430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.393872] Lustre: Skipped 1 previous similar message [ 6080.339426] Lustre: server umount lustre-OST0000 complete [ 6083.039831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6083.043256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.046921] Lustre: Skipped 1 previous similar message [ 6083.049265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.624275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.624752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.700293] Lustre: server umount lustre-MDT0000 complete [ 6088.941215] LustreError: 326751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469696 with bad export cookie 17297122839298630474 [ 6088.944701] LustreError: 326751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6088.944991] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6089.061372] Lustre: server umount lustre-MDT0001 complete [ 6091.905497] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:54:59 (1780469699) [ 6097.113344] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6100.456831] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6111.989939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.084429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6112.095602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6112.133480] Lustre: lustre-MDT0000: new disk, initializing [ 6112.157468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.163495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6113.478936] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6117.499288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.537354] Lustre: 332293:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6117.558528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6117.560273] Lustre: Skipped 1 previous similar message [ 6117.595340] Lustre: lustre-MDT0001: new disk, initializing [ 6117.626593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.638799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6117.642460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6118.905545] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6122.004871] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6123.176949] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6125.406816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.510020] Lustre: 334036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6126.516230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6126.518936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6126.554449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6127.360901] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6130.366828] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6141.921987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.922491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.924628] LustreError: Skipped 1 previous similar message [ 6141.925447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6141.928795] Lustre: Skipped 1 previous similar message [ 6141.930112] Lustre: Skipped 1 previous similar message [ 6145.486161] Lustre: server umount lustre-OST0000 complete [ 6146.729340] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6146.731817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.737266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.738947] Lustre: Skipped 1 previous similar message [ 6146.789851] Lustre: server umount lustre-MDT0000 complete [ 6148.000964] LustreError: 332286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780469755 with bad export cookie 17297122839298631356 [ 6148.003426] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6148.004151] LustreError: 332286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6154.159176] Lustre: server umount lustre-MDT0001 complete [ 6157.809264] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6158.875207] Key type lgssc unregistered [ 6159.005479] LNet: 336056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6159.007781] LNetError: 336056:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6159.016436] LNet: Removed LNI 192.168.202.160@tcp [ 6159.328702] Key type .llcrypt unregistered [ 6159.330392] Key type ._llcrypt unregistered [ 6191.098580] Key type ._llcrypt registered [ 6191.099603] Key type .llcrypt registered [ 6191.138518] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6191.235424] Key type .llcrypt unregistered [ 6191.236694] Key type ._llcrypt unregistered [ 6217.998682] Key type ._llcrypt registered [ 6217.999660] Key type .llcrypt registered [ 6218.036527] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6218.136196] Key type .llcrypt unregistered [ 6218.137469] Key type ._llcrypt unregistered [ 6229.483900] Key type ._llcrypt registered [ 6229.485013] Key type .llcrypt registered [ 6229.527111] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6229.630353] Key type .llcrypt unregistered [ 6229.632047] Key type ._llcrypt unregistered [ 6240.678615] Key type ._llcrypt registered [ 6240.679824] Key type .llcrypt registered [ 6240.716788] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6240.811199] Key type .llcrypt unregistered [ 6240.812178] Key type ._llcrypt unregistered [ 6268.538740] Key type ._llcrypt registered [ 6268.539942] Key type .llcrypt registered [ 6268.574407] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6268.666334] Key type .llcrypt unregistered [ 6268.667589] Key type ._llcrypt unregistered [ 6279.763275] Key type ._llcrypt registered [ 6279.764166] Key type .llcrypt registered [ 6279.802413] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6279.896389] Key type .llcrypt unregistered [ 6279.897670] Key type ._llcrypt unregistered [ 6290.914211] Key type ._llcrypt registered [ 6290.915486] Key type .llcrypt registered [ 6290.953620] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6291.052211] Key type .llcrypt unregistered [ 6291.053181] Key type ._llcrypt unregistered [ 6301.280078] Key type ._llcrypt registered [ 6301.281129] Key type .llcrypt registered [ 6301.315147] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6301.401242] Key type .llcrypt unregistered [ 6301.402165] Key type ._llcrypt unregistered [ 6312.003704] Key type ._llcrypt registered [ 6312.005109] Key type .llcrypt registered [ 6312.041872] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6312.127543] Key type .llcrypt unregistered [ 6312.128826] Key type ._llcrypt unregistered [ 6322.240271] Key type ._llcrypt registered [ 6322.241515] Key type .llcrypt registered [ 6322.275317] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6322.363367] Key type .llcrypt unregistered [ 6322.364244] Key type ._llcrypt unregistered [ 6342.804887] Key type ._llcrypt registered [ 6342.806009] Key type .llcrypt registered [ 6342.840951] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6342.934272] Key type .llcrypt unregistered [ 6342.935375] Key type ._llcrypt unregistered [ 6354.147744] Key type ._llcrypt registered [ 6354.148924] Key type .llcrypt registered [ 6354.184568] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6354.282363] Key type .llcrypt unregistered [ 6354.284292] Key type ._llcrypt unregistered [ 6365.556137] Key type ._llcrypt registered [ 6365.557119] Key type .llcrypt registered [ 6365.600567] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6365.705203] Key type .llcrypt unregistered [ 6365.706355] Key type ._llcrypt unregistered [ 6383.576429] Key type ._llcrypt registered [ 6383.577452] Key type .llcrypt registered [ 6383.613801] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6389.845280] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6390.268497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6390.296231] alg: No test for adler32 (adler32-zlib) [ 6391.153034] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6391.244286] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 6392.831226] Key type lgssc registered [ 6393.189513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.387144] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6404.392033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.482440] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6405.495491] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6405.530691] Lustre: tst250fs-MDT0000: new disk, initializing [ 6405.555390] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.561586] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6406.836824] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6410.985881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.023875] Lustre: 355702:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6411.047464] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6411.049360] Lustre: Skipped 1 previous similar message [ 6411.090301] Lustre: tst250fs-MDT0001: new disk, initializing [ 6411.123055] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6411.134162] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6411.137376] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6412.482142] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6414.774312] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6416.953974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.061079] Lustre: tst250fs-OST0000: new disk, initializing [ 6417.062712] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6417.065087] Lustre: 357299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6417.102122] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.964433] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6423.043090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6423.108179] Lustre: tst250fs-OST0001: new disk, initializing [ 6423.110949] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6423.115322] Lustre: 358143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6423.150089] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6423.225526] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6423.228963] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6423.292819] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6423.297993] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6425.113967] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6427.598214] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6438.883258] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.885329] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6438.888282] Lustre: Skipped 3 previous similar messages [ 6438.890176] Lustre: Skipped 2 previous similar messages [ 6444.004423] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.006612] Lustre: Skipped 1 previous similar message [ 6449.121403] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6449.123416] Lustre: Skipped 4 previous similar messages [ 6452.191126] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6452.260909] Lustre: server umount tst250fs-MDT0000 complete [ 6453.492703] LustreError: 355694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780470060 with bad export cookie 18095561011138040763 [ 6453.496288] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6453.496330] LustreError: 355694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6453.620458] Lustre: server umount tst250fs-MDT0001 complete [ 6465.095880] Lustre: server umount tst250fs-OST0000 complete [ 6476.577542] Lustre: server umount tst250fs-OST0001 complete [ 6482.456571] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6485.189668] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6495.188282] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:01:42 (1780470102) [ 6495.665176] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6500.735546] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6503.926144] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6514.618336] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6203 sec ========== 03:02:02 (1780470122) [ 6515.140066] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:02:02 (1780470122) === [ 6516.257441] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:02:03 (1780470123) === [ 6529.216253] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6530.226540] Key type lgssc unregistered [ 6530.352349] LNet: 367173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6530.354607] LNetError: 367173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6530.363260] LNet: Removed LNI 192.168.202.160@tcp [ 6530.640095] Key type .llcrypt unregistered [ 6530.641348] Key type ._llcrypt unregistered