[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498698759 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003222] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007295] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011087] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014089] *** VALIDATE selinux *** [ 0.022821] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026802] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029026] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030140] *** VALIDATE tmpfs *** [ 0.032066] *** VALIDATE proc *** [ 0.033254] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042211] debug: unmapping init [mem 0xffffffff9f059000-0xffffffff9f060fff] [ 0.044166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051020] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053323] rcu: Hierarchical SRCU implementation. [ 0.055356] smp: Bringing up secondary CPUs ... [ 0.056648] x86: Booting SMP configuration: [ 0.057035] .... node #0, CPUs: #1 #2 #3 [ 0.061136] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143312] node 0 deferred pages initialised in 77ms [ 0.146132] devtmpfs: initialized [ 0.147229] x86/mm: Memory block size: 128MB [ 0.149235] gcov: version magic: 0x41383552 [ 0.150590] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154144] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155260] pinctrl core: initialized pinctrl subsystem [ 0.156126] [ 0.156419] ************************************************************* [ 0.157009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158013] ** ** [ 0.159014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160015] ** ** [ 0.161012] ** This means that this kernel is built to expose internal ** [ 0.162015] ** IOMMU data structures, which may compromise security on ** [ 0.163009] ** your system. ** [ 0.164011] ** ** [ 0.165009] ** If you see this message and you are not debugging the ** [ 0.166009] ** kernel, report this immediately to your vendor! ** [ 0.167009] ** ** [ 0.168009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169010] ************************************************************* [ 0.170667] NET: Registered protocol family 16 [ 0.171296] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175015] cpuidle: using governor menu [ 0.177284] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178609] PCI: Using configuration type 1 for base access [ 0.179124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192131] cryptd: max_cpu_qlen set to 1000 [ 0.194203] ACPI: Added _OSI(Module Device) [ 0.196015] ACPI: Added _OSI(Processor Device) [ 0.197016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204044] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212351] ACPI: Interpreter enabled [ 0.213063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214010] ACPI: Using IOAPIC for interrupt routing [ 0.215098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216339] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225163] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230460] acpiphp: Slot [2] registered [ 0.231107] acpiphp: Slot [5] registered [ 0.232153] acpiphp: Slot [6] registered [ 0.233190] acpiphp: Slot [7] registered [ 0.235114] acpiphp: Slot [8] registered [ 0.236105] acpiphp: Slot [9] registered [ 0.238120] acpiphp: Slot [10] registered [ 0.239105] acpiphp: Slot [3] registered [ 0.240110] acpiphp: Slot [4] registered [ 0.242190] acpiphp: Slot [11] registered [ 0.243097] acpiphp: Slot [12] registered [ 0.245159] acpiphp: Slot [13] registered [ 0.246082] acpiphp: Slot [14] registered [ 0.247069] acpiphp: Slot [15] registered [ 0.248084] acpiphp: Slot [16] registered [ 0.250093] acpiphp: Slot [17] registered [ 0.251103] acpiphp: Slot [18] registered [ 0.253121] acpiphp: Slot [19] registered [ 0.255102] acpiphp: Slot [20] registered [ 0.256098] acpiphp: Slot [21] registered [ 0.257115] acpiphp: Slot [22] registered [ 0.259207] acpiphp: Slot [23] registered [ 0.260000] acpiphp: Slot [24] registered [ 0.260000] acpiphp: Slot [25] registered [ 0.261192] acpiphp: Slot [26] registered [ 0.262102] acpiphp: Slot [27] registered [ 0.264105] acpiphp: Slot [28] registered [ 0.265131] acpiphp: Slot [29] registered [ 0.267111] acpiphp: Slot [30] registered [ 0.268068] acpiphp: Slot [31] registered [ 0.269093] PCI host bridge to bus 0000:00 [ 0.271020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.285230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.286836] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.289163] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.303767] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.307017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.309017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.314669] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322053] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351087] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.399918] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.403047] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.410017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.424016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.435345] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.446024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.454016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.473018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.483154] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.490018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.498021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.512016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.521303] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.530018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.536016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.552018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.562702] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.568018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.573016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.587016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.598787] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.602598] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.604278] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.605209] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.607213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.611034] iommu: Default domain type: Passthrough [ 0.612324] SCSI subsystem initialized [ 0.614117] ACPI: bus type USB registered [ 0.615116] usbcore: registered new interface driver usbfs [ 0.616041] usbcore: registered new interface driver hub [ 0.617047] usbcore: registered new device driver usb [ 0.619184] pps_core: LinuxPPS API ver. 1 registered [ 0.621012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.624059] PTP clock support registered [ 0.626127] EDAC MC: Ver: 3.0.0 [ 0.628109] PCI: Using ACPI for IRQ routing [ 0.629615] NetLabel: Initializing [ 0.630011] NetLabel: domain hash size = 128 [ 0.632008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.634095] NetLabel: unlabeled traffic allowed by default [ 0.637060] vgaarb: loaded [ 0.638355] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.640018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.647000] clocksource: Switched to clocksource kvm-clock [ 0.760702] VFS: Disk quotas dquot_6.6.0 [ 0.761902] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.764289] *** VALIDATE ramfs *** [ 0.765776] *** VALIDATE hugetlbfs *** [ 0.767518] pnp: PnP ACPI init [ 0.769278] pnp: PnP ACPI: found 6 devices [ 0.782959] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.785271] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.786525] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.788133] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.789783] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.791215] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.792915] NET: Registered protocol family 2 [ 0.794555] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.798156] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.800656] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.804721] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.807219] TCP: Hash tables configured (established 65536 bind 65536) [ 0.809210] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.811640] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.813624] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816323] NET: Registered protocol family 1 [ 0.818295] RPC: Registered named UNIX socket transport module. [ 0.819883] RPC: Registered udp transport module. [ 0.821156] RPC: Registered tcp transport module. [ 0.822939] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.825360] NET: Registered protocol family 44 [ 0.827066] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.829326] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.831538] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.834174] PCI: CLS 0 bytes, default 64 [ 0.835379] Unpacking initramfs... [ 2.232663] debug: unmapping init [mem 0xffff9cba7cc54000-0xffff9cba7ffbffff] [ 2.236589] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.239016] software IO TLB: mapped [mem 0x00000000a2400000-0x00000000a6400000] (64MB) [ 2.241980] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.744438] Initialise system trusted keyrings [ 2.746245] Key type blacklist registered [ 2.748603] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.757216] zbud: loaded [ 2.759994] *** VALIDATE nfs *** [ 2.761312] *** VALIDATE nfs4 *** [ 2.762815] pstore: using deflate compression [ 2.765083] Platform Keyring initialized [ 2.855909] NET: Registered protocol family 38 [ 2.857050] Key type asymmetric registered [ 2.857947] Asymmetric key parser 'x509' registered [ 2.859152] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.861045] io scheduler mq-deadline registered [ 2.862070] io scheduler kyber registered [ 2.863102] io scheduler bfq registered [ 2.864257] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.865988] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.867848] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.869558] ACPI: Power Button [PWRF] [ 2.873239] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.877848] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.888523] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.894976] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.905158] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.931059] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.957495] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.961385] Non-volatile memory driver v1.3 [ 2.962407] Linux agpgart interface v0.103 [ 2.987314] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.989826] vda: detected capacity change from 0 to 76693504 [ 3.001279] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.004261] vdb: detected capacity change from 0 to 1073741824 [ 3.016116] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.018152] vdc: detected capacity change from 0 to 2621440000 [ 3.031109] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.034232] vdd: detected capacity change from 0 to 2621440000 [ 3.053544] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.056543] vde: detected capacity change from 0 to 4294967296 [ 3.090057] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.093206] vdf: detected capacity change from 0 to 4294967296 [ 3.099563] libphy: Fixed MDIO Bus: probed [ 3.104579] usbcore: registered new interface driver usbserial_generic [ 3.107279] usbserial: USB Serial support registered for generic [ 3.109839] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.114234] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.116249] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.118478] mousedev: PS/2 mouse device common for all mice [ 3.121370] rtc_cmos 00:05: RTC can wake from S4 [ 3.123938] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.125367] rtc_cmos 00:05: registered as rtc0 [ 3.130338] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.133653] intel_pstate: CPU model not supported [ 3.137544] hid: raw HID events driver (C) Jiri Kosina [ 3.139177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.140698] usbcore: registered new interface driver usbhid [ 3.140703] usbhid: USB HID core driver [ 3.141877] drop_monitor: Initializing network drop monitor service [ 3.153862] Initializing XFRM netlink socket [ 3.155861] NET: Registered protocol family 10 [ 3.158765] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.158891] Segment Routing with IPv6 [ 3.164399] NET: Registered protocol family 17 [ 3.166581] mpls_gso: MPLS GSO support [ 3.174800] RAS: Correctable Errors collector initialized. [ 3.177171] AVX version of gcm_enc/dec engaged. [ 3.179229] AES CTR mode by8 optimization enabled [ 3.254256] sched_clock: Marking stable (3254015816, 0)->(4230645851, -976630035) [ 3.256947] registered taskstats version 1 [ 3.258759] Loading compiled-in X.509 certificates [ 3.260251] zswap: loaded using pool lzo/zbud [ 3.284601] Key type big_key registered [ 3.297220] Key type encrypted registered [ 3.298640] ima: No TPM chip found, activating TPM-bypass! [ 3.300910] ima: Allocated hash algorithm: sha1 [ 3.302478] ima: No architecture policies found [ 3.304222] evm: Initialising EVM extended attributes: [ 3.305533] evm: security.selinux [ 3.306599] evm: security.ima [ 3.307569] evm: security.capability [ 3.309071] evm: HMAC attrs: 0x1 [ 3.311163] rtc_cmos 00:05: setting system clock to 2026-09-05 02:53:52 UTC (1788576832) [ 3.316176] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 3.318638] debug: unmapping init [mem 0xffffffff9ed82000-0xffffffff9f058fff] [ 3.330164] Write protecting the kernel read-only data: 28672k [ 3.333984] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 3.336638] debug: unmapping init [mem 0xffffffff9dd14000-0xffffffff9ddfffff] [ 3.371478] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.379948] systemd[1]: Detected virtualization kvm. [ 3.381934] systemd[1]: Detected architecture x86-64. [ 3.383623] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.411290] systemd[1]: No hostname configured. [ 3.413323] systemd[1]: Set hostname to . [ 3.415559] random: systemd: uninitialized urandom read (16 bytes read) [ 3.418330] systemd[1]: Initializing machine ID from random generator. [ 3.457159] random: ln: uninitialized urandom read (6 bytes read) [ 3.536559] random: systemd: uninitialized urandom read (16 bytes read) [ 3.538783] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.543947] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.547213] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.031441] device-mapper: uevent: version 1.0.3 [ 4.033808] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.686695] random: fast init done [ 4.722923] virtio_net virtio0 ens2: renamed from eth0 [ 4.802609] scsi host0: ata_piix [ 4.851181] scsi host1: ata_piix [ 4.852559] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.854885] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.409793] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.815108] random: crng init done [ 9.816560] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.066656] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.078476] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.302364] SELinux: Disabled at runtime. [ 11.364957] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.373571] systemd[1]: Detected virtualization kvm. [ 11.375514] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.735495] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.738537] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.742654] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.746126] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.749819] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.757180] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.762530] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [FAILED[ 11.836606] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.156075] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.425750] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.529086] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.595927] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.613897] EDAC sbridge: Ver: 1.1.2 [ 14.152659] Key type dns_resolver registered [ 14.459516] NFS: Registering the id_resolver key type [ 14.461528] Key type id_resolver registered [ 14.462812] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg252-server login: [ 40.754099] libcfs: loading out-of-tree module taints kernel. [ 40.775215] Key type ._llcrypt registered [ 40.776654] Key type .llcrypt registered [ 40.828147] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 49.026736] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 49.598299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.604943] alg: No test for adler32 (adler32-zlib) [ 50.614184] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 50.933065] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 52.575120] Key type lgssc registered [ 53.968687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.850351] hrtimer: interrupt took 4338679 ns [ 74.763722] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 121.217298] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 138.919876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.956792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.386661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.444383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.597896] Lustre: lustre-MDT0000: new disk, initializing [ 140.674331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.690455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.344484] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.396245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.565959] Lustre: 6498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.643093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.653404] Lustre: Skipped 1 previous similar message [ 161.774658] Lustre: lustre-MDT0001: new disk, initializing [ 161.867264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.900156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.916279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.412531] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.496613] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 182.069906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.342223] Lustre: lustre-OST0000: new disk, initializing [ 182.345611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 182.353780] Lustre: 8436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.415718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.687417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.704265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.788174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.487591] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.083732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.242973] Lustre: lustre-OST0001: new disk, initializing [ 204.253223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.262616] Lustre: 9507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.345415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.011508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.024784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.157557] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.762636] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.596957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.387232] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.115933] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.985402] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing yml_node [ 249.268459] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 252.105176] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 255.399141] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 257.088259] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 22:58:05 EDT 2026 [ 275.817720] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 277.648708] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.344356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.355498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.373078] Lustre: Skipped 3 previous similar messages [ 293.404294] Lustre: Skipped 3 previous similar messages [ 298.464297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.475031] Lustre: Skipped 3 previous similar messages [ 303.584197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.596237] Lustre: Skipped 3 previous similar messages [ 307.167308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 307.556903] Lustre: server umount lustre-MDT0000 complete [ 311.848609] LustreError: 7456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577141 with bad export cookie 6319408847783415212 [ 311.867239] LustreError: 7456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 311.867352] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.827027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.828403] LustreError: 6506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.845444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.858698] LustreError: 6506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 313.887962] Lustre: Skipped 1 previous similar message [ 316.841185] LustreError: 9515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.862950] LustreError: 9515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 318.528416] Lustre: server umount lustre-MDT0001 complete [ 323.122439] Lustre: server umount lustre-OST0000 complete [ 327.712987] Lustre: server umount lustre-OST0001 complete [ 335.862291] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 347.559780] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 386.117516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.481172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.531351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.656143] Lustre: lustre-MDT0000: new disk, initializing [ 386.746601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.776583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.549303] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 403.840444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.972324] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.088344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.092287] Lustre: Skipped 1 previous similar message [ 404.190601] Lustre: lustre-MDT0001: new disk, initializing [ 404.285478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.317221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.336483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.661420] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 419.794781] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.261054] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.778757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.185893] Lustre: lustre-OST0000: new disk, initializing [ 434.189308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.195317] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.335278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.155899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 436.177243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 436.316372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.191083] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 451.748911] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 456.073539] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 456.282677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.403560] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.590174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.915309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.918940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.933991] LustreError: Skipped 1 previous similar message [ 466.940046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.940053] Lustre: Skipped 2 previous similar messages [ 467.028919] Lustre: Skipped 2 previous similar messages [ 469.957878] Lustre: server umount lustre-OST0000 complete [ 475.621711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.631671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.646481] Lustre: Skipped 1 previous similar message [ 475.654885] Lustre: Skipped 2 previous similar messages [ 481.167526] Lustre: server umount lustre-MDT0000 complete [ 484.811437] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577314 with bad export cookie 6319408847783416745 [ 484.817607] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.828418] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 485.216160] Lustre: server umount lustre-MDT0001 complete [ 491.973109] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:01:59 (1788577319) [ 511.880236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.415107] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.483226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.603877] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.630758] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 522.731272] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.782053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.888523] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 540.166912] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 545.526978] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.614461] Lustre: Failing over lustre-MDT0000 [ 547.633350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.638702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.645147] Lustre: Skipped 1 previous similar message [ 547.661320] Lustre: Skipped 3 previous similar messages [ 547.843611] Lustre: server umount lustre-MDT0000 complete [ 552.802245] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.832429] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 567.138038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.295390] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.070155] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 572.706981] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 572.723299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 572.775760] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 581.784512] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 587.245270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.255145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 587.262206] Lustre: Skipped 1 previous similar message [ 587.282532] Lustre: Skipped 1 previous similar message [ 592.207813] Lustre: server umount lustre-MDT0000 complete [ 592.360826] LustreError: 20929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.380941] LustreError: 20929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 596.029570] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577425 with bad export cookie 6319408847783418159 [ 596.031516] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.035722] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 596.422953] Lustre: server umount lustre-MDT0001 complete [ 604.995041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.490568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.503154] Lustre: Skipped 2 previous similar messages [ 610.481570] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 619.184095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.536699] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 631.880478] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 637.638198] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 640.215717] Lustre: Failing over lustre-MDT0000 [ 640.440467] Lustre: server umount lustre-MDT0000 complete [ 641.002508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.009397] LustreError: 24244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.016106] Lustre: Skipped 1 previous similar message [ 641.055794] LustreError: 24244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 659.279967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.415645] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.783950] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 664.239914] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 664.810331] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 664.818552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 664.830333] Lustre: Skipped 1 previous similar message [ 664.870822] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 673.619201] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 677.701068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.736473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.744042] Lustre: Skipped 1 previous similar message [ 684.176584] Lustre: server umount lustre-MDT0000 complete [ 688.198384] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577517 with bad export cookie 6319408847783419097 [ 688.206899] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.209284] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 688.591133] Lustre: server umount lustre-MDT0001 complete [ 697.796343] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:05:25 (1788577525) [ 714.885290] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 725.675601] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 765.920604] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 776.822943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.068591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 777.090183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 777.157833] Lustre: lustre-MDT0000: new disk, initializing [ 777.239041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.243674] Lustre: Skipped 2 previous similar messages [ 777.256623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 781.908572] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 792.926740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.081643] Lustre: 29634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 793.456929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 798.459353] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 807.791716] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 814.140680] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 821.179439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 821.464677] Lustre: 31605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 822.657778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 822.717461] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 822.744686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 822.760533] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 828.562847] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 838.866724] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 844.078208] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 863.726648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 863.729263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.741652] LustreError: Skipped 1 previous similar message [ 863.743483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.766052] Lustre: Skipped 2 previous similar messages [ 863.789069] Lustre: Skipped 3 previous similar messages [ 868.806474] Lustre: server umount lustre-OST0000 complete [ 887.267466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 887.458130] Lustre: server umount lustre-MDT0000 complete [ 889.316075] LustreError: 30522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.331438] LustreError: 30522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 890.932705] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577720 with bad export cookie 6319408847783419503 [ 890.938241] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.950286] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 891.326610] Lustre: server umount lustre-MDT0001 complete [ 903.973330] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 907.039957] Key type lgssc unregistered [ 907.513538] LNet: 33583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.528182] LNetError: 33583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.550964] LNet: Removed LNI 192.168.202.152@tcp [ 908.509143] Key type .llcrypt unregistered [ 908.511029] Key type ._llcrypt unregistered [ 935.254355] Key type ._llcrypt registered [ 935.256673] Key type .llcrypt registered [ 935.346953] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 936.886204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.974908] alg: No test for adler32 (adler32-zlib) [ 938.029500] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 938.431411] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 940.143668] Key type lgssc registered [ 941.207869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.977128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.000910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.677892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.741297] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 959.972367] LustreError: 34992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.000664] LustreError: 34992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 962.650653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.153846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.834651] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 977.469677] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.046973] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 992.014313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.292561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.384141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 998.207580] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1006.200979] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1008.694610] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1025.570388] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:10:53 (1788577853) [ 1027.589306] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1029.466172] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1036.413181] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:11:04 (1788577864) [ 1046.247371] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:11:14 (1788577874) [ 1047.453024] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1056.279713] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1056.284458] Lustre: Skipped 2 previous similar messages [ 1058.111389] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:11:26 (1788577886) [ 1061.927873] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1061.932786] Lustre: Skipped 3 previous similar messages [ 1080.347720] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1080.359987] Lustre: Skipped 6 previous similar messages [ 1083.147451] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:11:51 (1788577911) [ 1086.080799] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1089.612620] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1097.943248] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:12:05 (1788577925) [ 1099.705946] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1099.711170] Lustre: Skipped 3 previous similar messages [ 1113.197824] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:12:21 (1788577941) [ 1128.425515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.430266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.449288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.503033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1131.514638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.515573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.550207] Lustre: Skipped 1 previous similar message [ 1132.964483] Lustre: server umount lustre-OST0000 complete [ 1137.120732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.131683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.147340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.732922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.735692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.315213] Lustre: server umount lustre-MDT0000 complete [ 1146.860527] LustreError: 35720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.200956] LustreError: 34973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577976 with bad export cookie 6049470589605336484 [ 1147.207654] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.216836] LustreError: 34973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1147.564702] Lustre: server umount lustre-MDT0001 complete [ 1158.400684] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1161.934468] Key type lgssc unregistered [ 1162.298757] LNet: 42852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.312110] LNetError: 42852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.339216] LNet: Removed LNI 192.168.202.152@tcp [ 1163.185905] Key type .llcrypt unregistered [ 1163.188097] Key type ._llcrypt unregistered [ 1180.868251] Key type ._llcrypt registered [ 1180.872040] Key type .llcrypt registered [ 1181.035704] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1197.104254] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1198.481678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.503585] alg: No test for adler32 (adler32-zlib) [ 1199.825965] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1200.065140] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1201.831163] Key type lgssc registered [ 1203.152186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1237.674273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1237.704821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.984711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1239.025691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1239.081401] Lustre: lustre-MDT0000: new disk, initializing [ 1239.157182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.181460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1243.720645] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1254.195978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.331634] Lustre: 46000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1254.419911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1254.427319] Lustre: Skipped 1 previous similar message [ 1254.491605] Lustre: lustre-MDT0001: new disk, initializing [ 1254.581926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.628233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1254.643753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1259.641878] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1269.593485] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1275.144803] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1282.977581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.310587] Lustre: lustre-OST0000: new disk, initializing [ 1283.319779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1283.332195] Lustre: 47972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1283.473484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.833443] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1292.391749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1292.405173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1292.494362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1300.514719] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.144894] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1308.546802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1314.878830] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1315.102769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1318.372258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1318.374566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.385745] LustreError: Skipped 1 previous similar message [ 1318.395572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.404606] Lustre: Skipped 1 previous similar message [ 1318.428541] Lustre: Skipped 1 previous similar message [ 1323.505411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1323.520572] Lustre: Skipped 1 previous similar message [ 1328.608704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1328.611977] Lustre: Skipped 1 previous similar message [ 1331.169778] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1331.549889] Lustre: server umount lustre-OST0000 complete [ 1336.802821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1336.815130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.840481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1338.858161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.968853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1343.973672] Lustre: Skipped 3 previous similar messages [ 1350.111249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1350.441133] Lustre: server umount lustre-MDT0000 complete [ 1354.920569] LustreError: 46798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578184 with bad export cookie 641544732770822963 [ 1354.925268] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.936223] LustreError: 46798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1355.247334] Lustre: server umount lustre-MDT0001 complete [ 1365.799926] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:16:33 (1788578193) [ 1372.350166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.887909] LustreError: 50434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.922283] LustreError: 50434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1373.007301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.519423] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1378.273269] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.394117] LustreError: 50434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.722554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.131059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.014793] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1400.657776] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1406.308145] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1413.716648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.113743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.775615] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1429.427621] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1437.721093] LustreError: 52963:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1438.936247] Lustre: Modifying parameter general.timeout=1 in log params [ 1439.968736] Lustre: Modifying parameter general.timeout=2 in log params [ 1442.210160] Lustre: Modifying parameter general.timeout=4 in log params [ 1442.213592] Lustre: Skipped 1 previous similar message [ 1446.306808] Lustre: Modifying parameter general.timeout=8 in log params [ 1446.313811] Lustre: Skipped 3 previous similar messages [ 1454.640195] Lustre: Modifying parameter general.timeout=17 in log params [ 1454.650854] Lustre: Skipped 8 previous similar messages [ 1471.003345] Lustre: Modifying parameter general.timeout=33 in log params [ 1471.005967] Lustre: Skipped 15 previous similar messages [ 1503.886609] Lustre: Modifying parameter general.timeout=65 in log params [ 1503.891657] Lustre: Skipped 31 previous similar messages [ 1568.146152] Lustre: Modifying parameter general.timeout=127 in log params [ 1568.151700] Lustre: Skipped 61 previous similar messages [ 1581.246409] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:20:09 (1788578409) [ 1584.260350] Lustre: 59918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1586.190087] Lustre: 60016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1588.228286] Lustre: 60114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1593.695293] Lustre: 60430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1593.702435] Lustre: 60430:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1622.334287] Lustre: 61793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1622.364254] LustreError: 61793:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1626.622490] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:20:54 (1788578454) [ 1641.959505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.969461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.982299] Lustre: Skipped 2 previous similar messages [ 1641.998357] Lustre: Skipped 4 previous similar messages [ 1646.207684] Lustre: server umount lustre-MDT0000 complete [ 1647.073480] LustreError: 51173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.095959] LustreError: 51173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1649.225736] LustreError: 53014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578478 with bad export cookie 641544732770823845 [ 1649.230052] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1649.232357] LustreError: 53014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1649.547371] Lustre: server umount lustre-MDT0001 complete [ 1663.087931] Lustre: server umount lustre-OST0000 complete [ 1690.030429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.394484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1690.689403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.107363] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1705.991250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.075656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1706.104587] Lustre: 64326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1711.139500] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1719.632893] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1725.991704] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1733.425643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.701419] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1734.044539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.055285] Lustre: Skipped 1 previous similar message [ 1740.970975] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1749.019520] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1751.089821] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1751.100161] Lustre: Skipped 26 previous similar messages [ 1751.194384] Lustre: 66892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1759.713950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1759.719122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.741070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.754179] Lustre: server umount lustre-OST0000 complete [ 1770.488823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.501319] Lustre: Skipped 2 previous similar messages [ 1777.883826] Lustre: server umount lustre-MDT0000 complete [ 1780.704177] LustreError: 65589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.828985] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578611 with bad export cookie 641544732770827044 [ 1781.846211] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1781.848813] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1782.220526] Lustre: server umount lustre-MDT0001 complete [ 1796.320397] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1799.379220] Key type lgssc unregistered [ 1799.598943] LNet: 68102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.607327] LNetError: 68102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.677300] LNet: Removed LNI 192.168.202.152@tcp [ 1801.525976] Key type .llcrypt unregistered [ 1801.527678] Key type ._llcrypt unregistered [ 1828.937902] Key type ._llcrypt registered [ 1828.947656] Key type .llcrypt registered [ 1829.067102] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1830.402261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.476656] alg: No test for adler32 (adler32-zlib) [ 1831.575831] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1831.899331] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1833.583193] Key type lgssc registered [ 1835.099130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1843.607511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1843.650751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.448059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.162632] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1854.948562] LustreError: 69516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.973416] LustreError: 69516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1860.069967] LustreError: 69515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.204473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.894262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1866.299945] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1874.872740] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.189697] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.081582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.915234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.327208] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1905.158412] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1913.908362] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1924.070209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.092213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.093172] Lustre: Skipped 1 previous similar message [ 1924.123710] Lustre: Skipped 2 previous similar messages [ 1929.189191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.197379] Lustre: Skipped 2 previous similar messages [ 1930.156323] Lustre: server umount lustre-MDT0000 complete [ 1933.800870] LustreError: 71366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578762 with bad export cookie 8768803558433290017 [ 1933.807088] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.811456] LustreError: 71366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1934.099374] Lustre: server umount lustre-MDT0001 complete [ 1948.417082] Lustre: server umount lustre-OST0000 complete [ 1981.206688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.484177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1981.526865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1981.791751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.956924] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1994.573535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.725482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1994.778943] Lustre: 74388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1995.343645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.571260] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2008.279926] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2015.172825] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2023.493155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2023.763553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2024.129592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.082892] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2038.998968] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2040.692394] Lustre: 76964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2042.719259] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2052.577973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2052.585922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.602810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.651670] Lustre: Skipped 2 previous similar messages [ 2054.662970] Lustre: server umount lustre-OST0000 complete [ 2061.794196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2061.806125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.838079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.847383] Lustre: Skipped 1 previous similar message [ 2065.526323] Lustre: server umount lustre-MDT0000 complete [ 2067.955652] LustreError: 74396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.985140] LustreError: 74396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2070.199753] LustreError: 74380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578899 with bad export cookie 8768803558433290857 [ 2070.204027] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.214664] LustreError: 74380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2070.582684] Lustre: server umount lustre-MDT0001 complete [ 2086.148246] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2091.163637] Key type lgssc unregistered [ 2091.541776] LNet: 78174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.548977] LNetError: 78174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.577502] LNet: Removed LNI 192.168.202.152@tcp [ 2092.587851] Key type .llcrypt unregistered [ 2092.590997] Key type ._llcrypt unregistered [ 2119.745516] Key type ._llcrypt registered [ 2119.747538] Key type .llcrypt registered [ 2119.845414] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2121.035524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.162256] alg: No test for adler32 (adler32-zlib) [ 2122.430103] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2122.753920] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2124.519171] Key type lgssc registered [ 2126.368828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.047467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2135.067814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.611663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.636941] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2146.793926] LustreError: 79585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.808407] LustreError: 79585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2150.994779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.637849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.944854] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2166.038555] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.367924] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.808715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2180.657859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.033546] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2195.576599] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2200.971525] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2201.818569] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2203.326691] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2203.332370] Lustre: Skipped 1 previous similar message [ 2205.996519] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2206.000808] Lustre: Skipped 2 previous similar messages [ 2210.785938] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2210.793750] Lustre: Skipped 5 previous similar messages [ 2219.551824] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2219.556639] Lustre: Skipped 8 previous similar messages [ 2236.510870] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2236.517529] Lustre: Skipped 17 previous similar messages [ 2268.644318] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2268.647517] Lustre: Skipped 31 previous similar messages [ 2307.661442] Lustre: 87691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2310.497566] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:32:18 (1788579138) [ 2312.136663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2313.980214] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:32:21 (1788579141) [ 2318.818075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2318.820764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.829621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2318.838828] Lustre: Skipped 1 previous similar message [ 2318.848121] Lustre: Skipped 1 previous similar message [ 2323.937458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.946121] Lustre: Skipped 1 previous similar message [ 2329.065380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.086207] Lustre: Skipped 1 previous similar message [ 2332.132206] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2332.331658] Lustre: server umount lustre-OST0000 complete [ 2339.305312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.315127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.321115] Lustre: Skipped 1 previous similar message [ 2339.336118] Lustre: Skipped 1 previous similar message [ 2342.949236] Lustre: server umount lustre-MDT0000 complete [ 2344.416872] LustreError: 79585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.427131] LustreError: 79585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2346.483778] LustreError: 79566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579175 with bad export cookie 6873190058161006133 [ 2346.499243] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.502354] LustreError: 79566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2346.785302] Lustre: server umount lustre-MDT0001 complete [ 2358.198171] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2361.516293] Key type lgssc unregistered [ 2361.859615] LNet: 88940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.868813] LNetError: 88940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.888556] LNet: Removed LNI 192.168.202.152@tcp [ 2362.690295] Key type .llcrypt unregistered [ 2362.694971] Key type ._llcrypt unregistered [ 2367.612821] Key type ._llcrypt registered [ 2367.618671] Key type .llcrypt registered [ 2367.736740] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2373.186369] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules [ 2374.538874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.649690] alg: No test for adler32 (adler32-zlib) [ 2375.864657] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2376.176705] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2377.904909] Key type lgssc registered [ 2379.460239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.207932] LustreError: 90200:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2400.422825] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2403.347873] LustreError: 90200:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2409.756861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2409.779083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.253257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.280908] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2421.728653] LustreError: 91205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.754202] LustreError: 91205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2425.581542] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:34:12 (1788579252) [ 2426.856575] LustreError: 91204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.937687] LustreError: 91236:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2431.952497] LustreError: 91236:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2431.964759] LustreError: 91236:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2432.364956] Lustre: server umount lustre-MDT0000 complete [ 2447.144557] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2452.395347] Key type lgssc unregistered [ 2452.860599] LNet: 92729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.882307] LNetError: 92729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.916654] LNet: Removed LNI 192.168.202.152@tcp [ 2454.200449] Key type .llcrypt unregistered [ 2454.205136] Key type ._llcrypt unregistered [ 2475.014823] Key type ._llcrypt registered [ 2475.020373] Key type .llcrypt registered [ 2475.133173] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2476.257709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.548399] alg: No test for adler32 (adler32-zlib) [ 2477.676940] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2478.003850] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2479.711202] Key type lgssc registered [ 2481.480377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2489.644251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2489.674301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.155398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.651633] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2501.601902] LustreError: 93942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.619398] LustreError: 93942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2506.174037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.037623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.007356] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2519.432124] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2525.526805] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2532.349801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.763188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.000236] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2548.022222] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2604.615324] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:37:12 (1788579432) [ 2607.114673] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2608.731273] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:37:16 (1788579436) [ 2614.240971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2614.257459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.276795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.829320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.836604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.848180] Lustre: Skipped 1 previous similar message [ 2619.653562] Lustre: server umount lustre-MDT0000 complete [ 2622.951685] LustreError: 97207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.993148] LustreError: 97207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2624.278030] LustreError: 93923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579453 with bad export cookie 16365200502168073159 [ 2624.291031] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.293631] LustreError: 93923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2624.893809] Lustre: server umount lustre-MDT0001 complete [ 2640.718827] Lustre: server umount lustre-OST0000 complete [ 2654.605482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.035880] LustreError: 98239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.294926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.017841] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2660.337874] LustreError: 98240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.441777] LustreError: 98239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.797711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.450878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.774635] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2681.998943] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2687.684895] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2702.764990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.858335] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2702.873766] LustreError: 98226:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2702.886542] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2702.891636] LustreError: 100322:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2702.893953] LustreError: 100322:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2702.896074] LustreError: 100322:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2703.003459] Lustre: server umount lustre-OST0000 complete [ 2703.008437] LustreError: 100322:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2713.027116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.585258] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2713.593185] LustreError: 98228:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2713.621875] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2713.657689] LustreError: 100790:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2713.671697] LustreError: 100790:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2713.709672] LustreError: 100790:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2713.860635] Lustre: server umount lustre-OST0000 complete [ 2713.867365] LustreError: 100790:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2725.194754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.508090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2725.525860] Lustre: Found index 0 for lustre-OST0000, updating log [ 2725.536253] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2725.685919] Lustre: lustre-OST0000: new disk, initializing [ 2725.690285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2725.718175] Lustre: 101327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2726.101437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.622820] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2740.123258] Lustre: Failing over lustre-OST0000 [ 2740.289595] Lustre: server umount lustre-OST0000 complete [ 2745.827286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2745.843184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.857257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.865133] Lustre: Skipped 2 previous similar messages [ 2749.923271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.926181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.980807] Lustre: Skipped 1 previous similar message [ 2751.289307] Lustre: server umount lustre-MDT0000 complete [ 2755.040736] LustreError: 101284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.103939] LustreError: 100343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579584 with bad export cookie 16365200502168075903 [ 2755.107662] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.111629] LustreError: 100343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2765.452785] Lustre: server umount lustre-MDT0001 complete [ 2775.155471] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:40:02 (1788579602) [ 2794.885517] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2807.119038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.905390] LustreError: 103674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.945073] LustreError: 103674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2808.199274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.600447] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2824.240403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.545712] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2834.271152] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2842.004262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2842.807266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.815264] Lustre: Skipped 1 previous similar message [ 2842.824621] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2848.928957] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2848.966640] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2848.969487] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2850.523985] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2861.605281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2861.745720] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2861.878862] Lustre: lustre-OST0001: new disk, initializing [ 2861.882562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2861.890086] Lustre: 105996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2870.587980] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2871.460311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2871.484837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2871.605187] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2880.072669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2887.623966] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2894.311763] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 2896.868608] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2896.878817] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.561258] Lustre: server umount lustre-MDT0001 complete [ 2899.379514] LustreError: 106479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.389872] LustreError: 106479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2909.439737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.566785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2909.570354] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2909.673797] Lustre: 103661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.445493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.456593] Lustre: Skipped 1 previous similar message [ 2911.597544] LustreError: 107514:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2911.615056] LustreError: 107514:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2911.648134] LustreError: 103667:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2911.992377] LustreError: 107528:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.152@tcp: cfg command failed: rc = -17 [ 2912.026538] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2912.026538] [ 2916.868487] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2917.073381] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2929.126527] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:42:36 (1788579756) [ 2957.023388] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2978.484378] Lustre: 109632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3005.210612] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3010.331789] Lustre: 110768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3101.152774] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.153278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.162749] Lustre: Skipped 3 previous similar messages [ 3101.184527] Lustre: Skipped 7 previous similar messages [ 3106.272124] LustreError: 103669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.309392] LustreError: 103669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3106.389387] Lustre: server umount lustre-MDT0000 complete [ 3109.984247] LustreError: 103655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579939 with bad export cookie 16365200502168076897 [ 3109.993570] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3110.005876] LustreError: 103655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3110.665940] Lustre: server umount lustre-MDT0001 complete [ 3125.486619] Lustre: server umount lustre-OST0000 complete [ 3127.202324] Lustre: 93252:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579940/real 1788579940] req@ffff9cbac45ea300 x1875461335799808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3127.232168] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.247580] Lustre: Skipped 2 previous similar messages [ 3129.820978] Lustre: server umount lustre-OST0001 complete [ 3136.462565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3147.864770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3157.733631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3168.494337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3192.403432] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3205.008685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.044903] Lustre: lustre-MDT0000: reset Object Index mappings [ 3205.989196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.989515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3221.716506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.769199] Lustre: lustre-MDT0001: reset Object Index mappings [ 3228.471489] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3231.995607] Lustre: 115869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3240.511965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.784223] LustreError: 116222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.809876] LustreError: 116222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3247.832266] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3248.978607] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3249.856291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3259.334118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3265.532408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3265.548583] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3268.578830] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3277.587283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3281.894210] Lustre: 117745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3336.885449] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:49:24 (1788580164) [ 3362.794250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.816466] Lustre: Skipped 2 previous similar messages [ 3362.826171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.856665] Lustre: Skipped 3 previous similar messages [ 3368.695843] Lustre: server umount lustre-MDT0000 complete [ 3374.557523] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580203 with bad export cookie 16365200502168123790 [ 3374.567996] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3374.569029] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3381.099684] Lustre: server umount lustre-MDT0001 complete [ 3392.380995] Lustre: server umount lustre-OST0000 complete [ 3403.155718] Lustre: server umount lustre-OST0001 complete [ 3412.018600] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3424.000497] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3468.399930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.665744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3468.701594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3468.835748] Lustre: lustre-MDT0000: new disk, initializing [ 3468.969636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.979937] Lustre: Skipped 3 previous similar messages [ 3469.005782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3474.554488] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3483.102833] Lustre: server umount lustre-MDT0000 complete [ 3493.305543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.752575] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3493.809705] Lustre: 122028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3493.840179] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3500.591215] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3506.299264] Lustre: server umount lustre-MDT0000 complete [ 3516.776287] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:52:24 (1788580344) [ 3518.934439] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3521.181188] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:52:28 (1788580348) [ 3542.323537] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3553.821931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.162477] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3568.458979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.565798] Lustre: 124080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3568.626215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3568.630842] Lustre: Skipped 1 previous similar message [ 3568.756753] Lustre: lustre-MDT0001: new disk, initializing [ 3568.909740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3568.928876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3573.515725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3578.427815] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3585.398157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3585.791159] Lustre: lustre-OST0000: new disk, initializing [ 3585.794090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3585.800502] Lustre: 125659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3587.305186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3587.317961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3587.478258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3594.175834] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3604.718421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.849493] Lustre: lustre-OST0001: new disk, initializing [ 3604.852992] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3604.859342] Lustre: 126531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3606.944295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3606.962540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3607.162520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3611.918458] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3622.231285] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3625.929270] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4005.989272] Lustre: 132945:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4006.006783] Lustre: 132945:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4006.022652] Lustre: 132945:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4006.030324] Lustre: 132945:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4006.045775] Lustre: 132945:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4006.056439] Lustre: 132945:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4266.133039] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:04:53 (1788581093) [ 4281.317186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4281.330976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.344665] Lustre: Skipped 3 previous similar messages [ 4281.350489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.360035] Lustre: Skipped 8 previous similar messages [ 4287.771567] Lustre: server umount lustre-MDT0000 complete [ 4287.974268] LustreError: 127334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.987286] LustreError: 127334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4292.203741] LustreError: 124073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581121 with bad export cookie 16365200502168191816 [ 4292.217513] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.224943] LustreError: 124073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4293.099350] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.116617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4293.117499] Lustre: Skipped 4 previous similar messages [ 4293.142165] Lustre: Skipped 5 previous similar messages [ 4299.016977] Lustre: server umount lustre-MDT0001 complete [ 4309.551467] Lustre: server umount lustre-OST0000 complete [ 4319.875373] Lustre: server umount lustre-OST0001 complete [ 4327.344960] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4337.716559] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4383.633945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.004656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4384.083899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4384.277560] Lustre: lustre-MDT0000: new disk, initializing [ 4384.411657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.426293] Lustre: Skipped 5 previous similar messages [ 4384.462975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4390.958325] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4402.042395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4402.219758] Lustre: 136010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4402.276213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4402.281082] Lustre: Skipped 1 previous similar message [ 4402.354381] Lustre: lustre-MDT0001: new disk, initializing [ 4402.510114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4402.523768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4407.612414] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4417.922206] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4424.841194] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4432.881831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4433.189111] Lustre: lustre-OST0000: new disk, initializing [ 4433.196132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4433.221179] Lustre: 137978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4434.357511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4434.369495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4434.488079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4441.381550] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4452.783844] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4463.358162] Lustre: lustre-MDD0000: changelog on [ 4465.904985] Lustre: lustre-MDD0001: changelog on [ 6198.915228] Lustre: lustre-MDD0001: changelog off [ 6201.210387] Lustre: 136016:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6202.017320] Lustre: lustre-MDD0000: changelog off [ 6209.011082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.026871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6209.040790] Lustre: Skipped 3 previous similar messages [ 6212.816418] Lustre: server umount lustre-OST0000 complete [ 6219.234155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6219.237368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6219.252361] Lustre: Skipped 2 previous similar messages [ 6223.289393] Lustre: server umount lustre-MDT0000 complete [ 6224.358075] LustreError: 136021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.376349] LustreError: 136021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6227.065132] LustreError: 136002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583056 with bad export cookie 16365200502168966261 [ 6227.073399] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.080793] LustreError: 136002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6227.573738] Lustre: server umount lustre-MDT0001 complete [ 6238.591459] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6241.316988] Key type lgssc unregistered [ 6241.731050] LNet: 147096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6241.752458] LNetError: 147096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6242.793048] LNet: Removed LNI 192.168.202.152@tcp [ 6243.667145] Key type .llcrypt unregistered [ 6243.668533] Key type ._llcrypt unregistered [ 6264.264728] Key type ._llcrypt registered [ 6264.268699] Key type .llcrypt registered [ 6264.347345] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6279.560411] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6280.659401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6280.779267] alg: No test for adler32 (adler32-zlib) [ 6281.927354] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6282.317863] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6284.008299] Key type lgssc registered [ 6285.037634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6323.363705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6323.398959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.733571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6324.767864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6324.882787] Lustre: lustre-MDT0000: new disk, initializing [ 6325.103270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.116419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6331.165245] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6342.075846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.152808] Lustre: 150294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6342.190989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6342.194394] Lustre: Skipped 1 previous similar message [ 6342.243443] Lustre: lustre-MDT0001: new disk, initializing [ 6342.298103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6342.338046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6342.348927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6346.667873] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6356.134924] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6362.147581] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6369.634135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.174801] Lustre: lustre-OST0000: new disk, initializing [ 6370.177717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6370.181482] Lustre: 152266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6370.307116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.389371] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6379.057660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6379.066352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6379.165207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6387.861611] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6395.183761] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_module obdecho/obdecho [ 6397.036566] LustreError: 153269:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6397.042802] LustreError: 153269:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6398.808309] LustreError: 153365:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6409.184660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6409.201581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.239902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.517883] Lustre: server umount lustre-MDT0000 complete [ 6418.853235] LustreError: 150286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583248 with bad export cookie 13013708650691739382 [ 6418.854998] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6418.878948] LustreError: 150286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6419.231214] Lustre: server umount lustre-MDT0001 complete [ 6433.978641] Lustre: server umount lustre-OST0000 complete [ 6458.018838] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6470.461504] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6502.351799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.646240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6502.696642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6502.799420] Lustre: lustre-MDT0000: new disk, initializing [ 6502.849628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.862347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6507.381460] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6518.954121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.105955] Lustre: 157085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6519.163616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6519.167526] Lustre: Skipped 1 previous similar message [ 6519.237883] Lustre: lustre-MDT0001: new disk, initializing [ 6519.319143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6519.364993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6519.379852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6521.970679] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6531.301673] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6536.902380] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.450956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.771732] Lustre: lustre-OST0000: new disk, initializing [ 6544.779740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6544.786346] Lustre: 159051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6544.869994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.246662] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6551.698084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6551.706779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6551.833266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6561.283319] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6566.269827] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6566.447549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6571.445604] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6571.586451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6577.633954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6577.637731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.660098] LustreError: Skipped 1 previous similar message [ 6577.678075] Lustre: Skipped 2 previous similar messages [ 6577.685219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6577.697039] Lustre: Skipped 3 previous similar messages [ 6579.569766] Lustre: server umount lustre-OST0000 complete [ 6585.825358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6585.849261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.874141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6587.874402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6587.876908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.029166] Lustre: server umount lustre-MDT0000 complete [ 6592.998609] LustreError: 158094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6593.019940] LustreError: 158094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6594.646962] LustreError: 157075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583423 with bad export cookie 13013708650691740334 [ 6594.650706] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6594.664320] LustreError: 157075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6595.163803] Lustre: server umount lustre-MDT0001 complete [ 6599.064935] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:43:46 (1788583426) [ 6600.854707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6602.754638] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:43:50 (1788583430) [ 6611.369729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.044067] LustreError: 161462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6612.139455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.871738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6617.580649] LustreError: 161463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.692580] LustreError: 161462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.782159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.375796] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6638.452962] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6644.658728] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6652.277270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6652.653652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.661988] Lustre: Skipped 1 previous similar message [ 6659.359783] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6667.178889] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6678.472071] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 6678.485737] Lustre: Skipped 1 previous similar message [ 6679.414343] Lustre: server umount lustre-MDT0000 complete [ 6681.573958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.576919] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6681.600769] Lustre: Skipped 2 previous similar messages [ 6681.615222] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6683.427612] LustreError: 161443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583512 with bad export cookie 13013708650691741216 [ 6683.438817] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6683.446830] LustreError: 161443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6686.691604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6686.705809] Lustre: Skipped 1 previous similar message [ 6690.073560] Lustre: server umount lustre-MDT0001 complete [ 6697.200786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6711.661862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.857698] LustreError: 165303:0:(llog.c:1655:llog_backup()) MGC192.168.202.152@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6711.869742] Lustre: 165303:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.152@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6713.313401] LustreError: 165308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6713.322946] LustreError: 165308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6730.678558] LustreError: 165309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.704622] LustreError: 165309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6764.005024] LustreError: 163295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.024683] LustreError: 163295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 6772.703321] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6772.717390] Lustre: 165303:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6813.695252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6818.736259] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6818.791199] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6828.014066] LustreError: 165339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.054385] LustreError: 165339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 6828.336858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.368619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6853.604090] LustreError: 148516:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006a041f04 x1875465324335488/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6858.871342] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6859.241478] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6866.717878] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6868.056585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6873.644958] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6875.556462] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6895.072535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6895.074980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.080723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6895.080734] Lustre: Skipped 2 previous similar messages [ 6895.112425] Lustre: Skipped 3 previous similar messages [ 6900.071326] Lustre: server umount lustre-MDT0000 complete [ 6904.405461] LustreError: 161443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583733 with bad export cookie 13013708650691747684 [ 6904.425951] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6904.751629] Lustre: server umount lustre-MDT0001 complete [ 6919.449123] Lustre: server umount lustre-OST0000 complete [ 6942.649236] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6955.539856] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6995.142978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.506813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.540059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6995.661272] Lustre: lustre-MDT0000: new disk, initializing [ 6995.718624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.728590] Lustre: Skipped 1 previous similar message [ 6995.741562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7000.376322] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7011.911166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.044263] Lustre: 170412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7012.521331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7017.683658] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7028.025346] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7034.526496] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7041.313654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.612591] Lustre: lustre-OST0000: new disk, initializing [ 7041.616431] Lustre: Skipped 1 previous similar message [ 7041.619931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7041.626018] Lustre: Skipped 2 previous similar messages [ 7041.633903] Lustre: 172379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7048.379909] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7050.854589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7050.874038] Lustre: Skipped 1 previous similar message [ 7050.876133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7050.969040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7058.310887] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7063.648199] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7063.863184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7069.332896] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7069.586133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7076.834137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.836178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.838791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7076.838800] Lustre: Skipped 2 previous similar messages [ 7076.858090] Lustre: Skipped 1 previous similar message [ 7076.877648] LustreError: Skipped 1 previous similar message [ 7086.047733] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7086.232683] Lustre: server umount lustre-OST0000 complete [ 7096.569556] Lustre: server umount lustre-MDT0000 complete [ 7097.320352] LustreError: 170424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.358283] LustreError: 170424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 7100.351835] LustreError: 170405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583929 with bad export cookie 13013708650691764512 [ 7100.357642] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7100.358623] LustreError: 170405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7100.701828] Lustre: server umount lustre-MDT0001 complete [ 7109.569854] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:52:17 (1788583937) [ 7116.462534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.345515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7129.718963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.426261] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7142.523931] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7148.124155] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7155.510214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7162.527389] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7170.042383] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7179.682423] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:53:27 (1788584007) [ 7186.912415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7186.920233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.927548] Lustre: Skipped 2 previous similar messages [ 7186.938339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7186.949637] Lustre: Skipped 5 previous similar messages [ 7190.611536] Lustre: server umount lustre-OST0000 complete [ 7200.601744] Lustre: server umount lustre-MDT0000 complete [ 7204.473247] LustreError: 174773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584033 with bad export cookie 13013708650691765394 [ 7204.488850] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.931191] Lustre: server umount lustre-MDT0001 complete [ 7216.428335] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7219.076786] Key type lgssc unregistered [ 7219.307690] LNet: 178521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7219.312581] LNetError: 178521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7219.326572] LNet: Removed LNI 192.168.202.152@tcp [ 7220.150551] Key type .llcrypt unregistered [ 7220.152776] Key type ._llcrypt unregistered [ 7238.524147] Key type ._llcrypt registered [ 7238.527674] Key type .llcrypt registered [ 7238.662370] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7240.332855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7240.392130] alg: No test for adler32 (adler32-zlib) [ 7241.444241] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7241.676534] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7243.336143] Key type lgssc registered [ 7244.422975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7251.483561] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7251.498204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7267.040898] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7272.159556] LustreError: 179765:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 7277.279970] LustreError: 179765:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7277.295907] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7277.306223] LustreError: MGC192.168.202.152@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7277.318708] LustreError: 179765:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7277.349872] LustreError: 179765:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7277.359993] LustreError: 179765:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7277.440238] Lustre: server umount lustre-OST0000 complete [ 7277.450032] LustreError: 179765:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7286.135072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7287.639355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7292.331317] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7296.993728] LustreError: 180264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.000327] LustreError: 180264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7300.295391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.597954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7305.689073] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7312.917860] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7318.321779] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7324.695757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.903931] LustreError: 182083:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7324.915292] Lustre: 182083:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.152@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7325.086055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7331.564912] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7338.958460] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7341.537927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7341.551987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.565869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.562661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.576687] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7342.598140] Lustre: Skipped 1 previous similar message [ 7342.627631] Lustre: Skipped 1 previous similar message [ 7347.561959] Lustre: server umount lustre-MDT0000 complete [ 7347.681091] LustreError: 181248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.687416] LustreError: 181248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7351.242438] LustreError: 180246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584180 with bad export cookie 12384603103326128471 [ 7351.258728] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7351.262160] LustreError: 180246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7351.794353] Lustre: server umount lustre-MDT0001 complete [ 7365.683488] Lustre: server umount lustre-OST0000 complete [ 7375.404473] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7388.117442] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7426.757258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.991166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7427.024454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7427.190768] Lustre: lustre-MDT0000: new disk, initializing [ 7427.299171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.317843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7432.645552] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7444.008192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7444.144406] Lustre: 185605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7444.202431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7444.206369] Lustre: Skipped 1 previous similar message [ 7444.278375] Lustre: lustre-MDT0001: new disk, initializing [ 7444.375902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7444.444890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7444.460601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7449.216593] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7460.450692] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7466.917607] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7474.679538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7474.987540] Lustre: lustre-OST0000: new disk, initializing [ 7474.994789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7475.002235] Lustre: 187576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7475.116609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.570637] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7484.496929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7484.505718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7484.645491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7492.178979] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7496.807687] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7497.050345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7501.828500] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7502.054034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7505.389646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7505.389727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.397099] LustreError: Skipped 1 previous similar message [ 7505.399065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7505.412075] Lustre: Skipped 1 previous similar message [ 7505.422388] Lustre: Skipped 1 previous similar message [ 7510.497097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7510.515186] Lustre: Skipped 1 previous similar message [ 7511.005184] Lustre: server umount lustre-OST0000 complete [ 7515.617348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.640805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.659859] Lustre: Skipped 1 previous similar message [ 7515.696191] Lustre: Skipped 2 previous similar messages [ 7522.114237] Lustre: server umount lustre-MDT0000 complete [ 7525.275486] LustreError: 185599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584354 with bad export cookie 12384603103326129185 [ 7525.281963] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.550776] Lustre: server umount lustre-MDT0001 complete [ 7535.249068] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:59:23 (1788584363) [ 7554.710958] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7558.477621] Key type lgssc unregistered [ 7559.022792] LNet: 190522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7559.032127] LNetError: 190522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7560.045567] LNet: Removed LNI 192.168.202.152@tcp [ 7561.353617] Key type .llcrypt unregistered [ 7561.356611] Key type ._llcrypt unregistered [ 7582.957462] Key type ._llcrypt registered [ 7582.959816] Key type .llcrypt registered [ 7583.060444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7584.067421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7584.090544] alg: No test for adler32 (adler32-zlib) [ 7585.114996] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7585.259870] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7586.959198] Key type lgssc registered [ 7588.151838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7596.195338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7596.220768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.760725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7602.669851] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7607.268450] LustreError: 191739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.280462] LustreError: 191739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7611.716067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.067447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7616.386923] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7624.668783] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7630.654668] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7639.743765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7639.993889] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7639.998591] LustreError: 193618:0:(llog.c:1696:llog_backup()) MGC192.168.202.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7640.004645] Lustre: 193618:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.152@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7640.160726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.726523] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7654.642716] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7657.959033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.965414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7657.994906] Lustre: Skipped 1 previous similar message [ 7658.019407] Lustre: Skipped 2 previous similar messages [ 7659.047155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.053669] Lustre: Skipped 2 previous similar messages [ 7663.637306] Lustre: server umount lustre-MDT0000 complete [ 7664.105369] LustreError: 191739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.127458] LustreError: 191739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7667.338064] LustreError: 191718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584496 with bad export cookie 677443619666818640 [ 7667.350475] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7667.360635] LustreError: 191718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7669.216503] LustreError: 193643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.217554] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.253617] Lustre: Skipped 1 previous similar message [ 7669.265061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7669.343789] LustreError: 192469:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000016dbed6d x1875466690523392/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7669.539112] Lustre: server umount lustre-MDT0001 complete [ 7673.014715] Lustre: server umount lustre-OST0000 complete [ 7682.276762] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7691.842731] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7731.302948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.644023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7731.668954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7731.740309] Lustre: lustre-MDT0000: new disk, initializing [ 7731.855196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7731.874884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7736.588223] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7748.819549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7748.960644] Lustre: 197142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7749.023234] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7749.027406] Lustre: Skipped 1 previous similar message [ 7749.030524] LustreError: 197924:0:(llog.c:1696:llog_backup()) MGC192.168.202.152@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7749.037586] LustreError: 197924:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7749.043616] Lustre: 197924:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.152@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7749.062862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7749.069213] Lustre: Skipped 1 previous similar message [ 7749.204679] Lustre: lustre-MDT0001: new disk, initializing [ 7749.386149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7749.444073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7749.463407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7754.638953] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7765.437882] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7771.739236] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7779.610897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.972778] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7779.976920] Lustre: Skipped 1 previous similar message [ 7779.980204] LustreError: 199086:0:(llog.c:1696:llog_backup()) MGC192.168.202.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7779.990012] LustreError: 199086:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7780.004828] Lustre: 199086:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.152@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7780.016925] Lustre: 199086:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7780.042966] Lustre: lustre-OST0000: new disk, initializing [ 7780.046172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7780.053110] Lustre: 199109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7780.185483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7786.849996] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7787.087397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7787.103863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7787.278161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7797.225556] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7802.970703] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7803.341362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7808.870360] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7809.142614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7813.089530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.096159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.119334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.138518] Lustre: Skipped 1 previous similar message [ 7818.219455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7818.229321] Lustre: Skipped 1 previous similar message [ 7825.375070] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7825.641388] Lustre: server umount lustre-OST0000 complete [ 7831.521386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7831.528957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.541200] Lustre: Skipped 1 previous similar message [ 7831.545428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.552774] Lustre: Skipped 2 previous similar messages [ 7836.994664] Lustre: server umount lustre-MDT0000 complete [ 7838.694785] LustreError: 197153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.719279] LustreError: 197153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7841.873031] LustreError: 197132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584671 with bad export cookie 677443619666819368 [ 7841.886811] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7841.887120] LustreError: 197132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7842.687170] Lustre: server umount lustre-MDT0001 complete [ 7852.158326] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:04:40 (1788584680) [ 7870.858886] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7882.083619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.562053] LustreError: 202242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.624058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7887.318495] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7891.954634] LustreError: 202242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.987455] LustreError: 202242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7897.342500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.923615] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7906.413494] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7913.266367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7919.971535] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7930.800349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.056163] Lustre: lustre-OST0001: new disk, initializing [ 7931.060577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7931.067134] Lustre: 204551:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7931.162990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7931.170237] Lustre: Skipped 2 previous similar messages [ 7937.945132] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7939.693643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7939.699808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7939.795208] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7949.136457] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7959.869929] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7961.872716] Lustre: DEBUG MARKER: ADD OST3 [ 7968.323489] loop: module loaded [ 7968.455780] loop0: detected capacity change from 0 to 204800000 [ 7968.760024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7968.771066] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7968.783340] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7975.385726] loop0: detected capacity change from 0 to 204800000 [ 7976.120907] Lustre: DEBUG MARKER: STOP OST3 [ 7982.494437] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7982.892301] Lustre: lustre-OST0003: new disk, initializing [ 7982.897535] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7982.916289] Lustre: 206588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7984.881598] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7985.112647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7985.118551] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7985.160666] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7986.689559] Lustre: Failing over lustre-OST0003 [ 7986.939708] LustreError: 206722:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7987.007948] Lustre: server umount lustre-OST0003 complete [ 7989.706170] LustreError: 204374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.716140] LustreError: 204374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7990.257416] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7990.258075] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.282491] Lustre: Skipped 2 previous similar messages [ 7990.649286] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7999.746586] Lustre: DEBUG MARKER: START OST3 again [ 8008.105643] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8008.647647] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8008.650577] Lustre: Skipped 1 previous similar message [ 8008.673785] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8009.978312] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8010.222466] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8010.222895] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8017.562496] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8039.987052] Lustre: Failing over lustre-OST0003 [ 8040.114593] Lustre: server umount lustre-OST0003 complete [ 8040.931422] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8040.944957] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.971990] LustreError: 204374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.989540] LustreError: 204374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8059.879838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.893692] Lustre: Skipped 4 previous similar messages [ 8059.894423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.907898] Lustre: Skipped 5 previous similar messages [ 8065.165266] Lustre: server umount lustre-MDT0000 complete [ 8069.471517] LustreError: 202223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584898 with bad export cookie 677443619666820250 [ 8069.481538] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8069.500912] LustreError: 202223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8076.000851] Lustre: server umount lustre-MDT0001 complete [ 8080.122436] Lustre: server umount lustre-OST0000 complete [ 8084.839505] Lustre: server umount lustre-OST0001 complete [ 8093.629371] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8103.044156] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8147.030273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.255712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8147.278247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8147.360456] Lustre: lustre-MDT0000: new disk, initializing [ 8147.451639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8147.470727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8152.616454] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8166.720447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8166.852099] Lustre: 212007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8166.989577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8166.998043] Lustre: Skipped 1 previous similar message [ 8167.164425] Lustre: lustre-MDT0001: new disk, initializing [ 8167.364170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8167.384815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8172.597602] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8183.996753] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8190.809834] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8198.562437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8198.962810] Lustre: 213978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8200.478438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8200.491690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8200.639434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8206.291923] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8215.730513] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8220.659193] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8220.952713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8226.373297] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8226.724610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8231.406786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8231.415657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8231.423513] Lustre: Skipped 2 previous similar messages [ 8231.425193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8231.425201] Lustre: Skipped 8 previous similar messages [ 8231.454629] LustreError: Skipped 2 previous similar messages [ 8243.170663] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8243.504608] Lustre: server umount lustre-OST0000 complete [ 8249.313678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8262.111162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8262.412396] Lustre: server umount lustre-MDT0000 complete [ 8267.245040] LustreError: 212020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.291596] LustreError: 212020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 8267.500094] LustreError: 212000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585096 with bad export cookie 677443619666822546 [ 8267.504719] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8267.524147] LustreError: 212000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8268.253484] Lustre: server umount lustre-MDT0001 complete [ 8273.066338] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:11:40 (1788585100) [ 8292.996027] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8306.160858] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8351.945983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.396396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8352.420727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8352.424482] Lustre: Skipped 1 previous similar message [ 8352.491502] Lustre: lustre-MDT0000: new disk, initializing [ 8352.493619] Lustre: Skipped 1 previous similar message [ 8352.561585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8357.138891] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8367.391393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.586226] Lustre: 218594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8367.869758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8372.428241] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8381.473904] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8388.419131] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8396.867266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8397.071938] Lustre: 220563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8398.593531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8404.155714] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8415.306142] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8421.286305] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8421.547837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8426.554709] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8426.755952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8429.537792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8429.539982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.555594] LustreError: Skipped 1 previous similar message [ 8429.559920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8429.559928] Lustre: Skipped 12 previous similar messages [ 8429.598881] Lustre: Skipped 4 previous similar messages [ 8442.847351] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8443.137863] Lustre: server umount lustre-OST0000 complete [ 8454.164879] Lustre: server umount lustre-MDT0000 complete [ 8455.141834] LustreError: 219964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.174862] LustreError: 219964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8459.087063] LustreError: 219388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585288 with bad export cookie 677443619666823428 [ 8459.096207] LustreError: 219388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8459.096470] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.447622] Lustre: server umount lustre-MDT0001 complete [ 8468.677646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.400917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.404048] Lustre: Skipped 5 previous similar messages [ 8475.481322] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8485.167479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.122419] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8498.461688] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.223568] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.012517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8523.117922] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8531.532673] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8549.350899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8551.121372] Lustre: server umount lustre-OST0000 complete [ 8559.584711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.584844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.597893] Lustre: Skipped 5 previous similar messages [ 8559.646699] Lustre: Skipped 11 previous similar messages [ 8562.208315] Lustre: server umount lustre-MDT0000 complete [ 8566.603683] LustreError: 223517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585395 with bad export cookie 677443619666824303 [ 8566.620688] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8566.628926] LustreError: 223517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8567.143767] Lustre: server umount lustre-MDT0001 complete [ 8669.245670] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8672.965993] Key type lgssc unregistered [ 8673.363995] LNet: 226467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8673.369840] LNetError: 226467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8674.410754] LNet: Removed LNI 192.168.202.152@tcp [ 8675.382350] Key type .llcrypt unregistered [ 8675.385419] Key type ._llcrypt unregistered [ 8703.584668] Key type ._llcrypt registered [ 8703.587971] Key type .llcrypt registered [ 8703.719908] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8722.595603] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8724.310588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8724.445335] alg: No test for adler32 (adler32-zlib) [ 8725.585729] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8726.025625] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8727.735166] Key type lgssc registered [ 8729.167360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8766.133773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8766.192734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.502874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8767.535485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8767.627257] Lustre: lustre-MDT0000: new disk, initializing [ 8767.694309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8767.711412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8772.045405] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8782.133627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.264581] Lustre: 229815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8782.326380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8782.330279] Lustre: Skipped 1 previous similar message [ 8782.385096] Lustre: lustre-MDT0001: new disk, initializing [ 8782.473623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8782.548383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8782.584326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8787.511344] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8797.717557] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8804.134304] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8811.600038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8811.980529] Lustre: lustre-OST0000: new disk, initializing [ 8811.986610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8811.995187] Lustre: 231783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8812.119713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.795839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8818.810639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8818.957352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8820.047445] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8830.660285] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8836.078996] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8836.371783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8841.785319] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8842.085518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8844.771930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.780120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8844.801592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.895660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.903171] Lustre: Skipped 2 previous similar messages [ 8850.306424] Lustre: server umount lustre-OST0000 complete [ 8855.011025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.019459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8855.042324] Lustre: Skipped 2 previous similar messages [ 8855.058338] Lustre: Skipped 1 previous similar message [ 8860.131673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.141238] Lustre: Skipped 1 previous similar message [ 8860.840542] Lustre: server umount lustre-MDT0000 complete [ 8864.742942] LustreError: 229809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585693 with bad export cookie 6204249795705560970 [ 8864.748269] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8864.758605] LustreError: 229809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8865.083229] Lustre: server umount lustre-MDT0001 complete [ 8884.356842] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8894.585705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.926187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.566454] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8900.067887] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8900.097873] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8905.185350] LustreError: 234720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.495233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.825191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8912.657402] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8915.923778] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8922.962094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.397465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.268948] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8940.610804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.911126] Lustre: lustre-OST0001: new disk, initializing [ 8940.922098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8940.937812] Lustre: 237026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8941.055877] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8946.763230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8946.778555] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8946.900512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8947.889730] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8958.473944] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8967.860097] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8984.932974] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:23:32 (1788585812) [ 8988.130523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.145794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8988.148313] Lustre: Skipped 3 previous similar messages [ 8988.161754] Lustre: Skipped 3 previous similar messages [ 8994.029716] Lustre: server umount lustre-MDT0000 complete [ 8997.864976] LustreError: 234701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585827 with bad export cookie 6204249795705561859 [ 8997.872636] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8997.879056] LustreError: 234701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8998.260914] Lustre: server umount lustre-MDT0001 complete [ 9012.678335] Lustre: server umount lustre-OST0000 complete [ 9014.495208] Lustre: 228039:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585827/real 1788585827] req@ffff9cb9ca525880 x1875467887033984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9014.533122] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.045299] Lustre: server umount lustre-OST0001 complete [ 9024.498027] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9037.345974] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9077.966814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.201838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.232314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9078.395450] Lustre: lustre-MDT0000: new disk, initializing [ 9078.468408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9078.515909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9082.075773] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9095.216669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.323226] Lustre: 240974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9095.366841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9095.371134] Lustre: Skipped 1 previous similar message [ 9095.431517] Lustre: lustre-MDT0001: new disk, initializing [ 9095.549545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9095.562787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9100.537443] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9110.493310] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9117.297965] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9125.564130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9125.898637] Lustre: lustre-OST0000: new disk, initializing [ 9125.908659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9125.920853] Lustre: 242945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9127.402181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9127.416523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9127.637597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9133.731765] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9144.570894] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9150.035367] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9150.358367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9155.850372] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9156.065923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9158.626072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9158.632701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.648544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9158.658082] Lustre: Skipped 4 previous similar messages [ 9163.237258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9164.381416] Lustre: server umount lustre-OST0000 complete [ 9168.864667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.886314] Lustre: Skipped 2 previous similar messages [ 9174.833497] Lustre: server umount lustre-MDT0000 complete [ 9179.106606] LustreError: 240987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.144631] LustreError: 240987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9179.246976] LustreError: 240968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586008 with bad export cookie 6204249795705563994 [ 9179.253553] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9179.255127] LustreError: 240968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9179.593843] Lustre: server umount lustre-MDT0001 complete [ 9189.125323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.679946] LustreError: 245163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.774346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9189.790024] Lustre: Skipped 2 previous similar messages [ 9194.863974] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9194.980611] LustreError: 245164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.196541] LustreError: 245164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.218944] LustreError: 245164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9204.743702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.134313] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9219.867173] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9225.855674] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9233.375182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.169945] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9251.862160] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9397.729820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9397.746412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.767466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9397.775519] Lustre: Skipped 6 previous similar messages [ 9403.329890] Lustre: server umount lustre-OST0000 complete [ 9413.943785] Lustre: server umount lustre-MDT0000 complete [ 9414.113535] LustreError: 245892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.144206] LustreError: 245892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9418.188618] LustreError: 245145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586247 with bad export cookie 6204249795705564876 [ 9418.195627] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9418.199607] LustreError: 245145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9418.667271] Lustre: server umount lustre-MDT0001 complete [ 9522.179706] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9525.588882] Key type lgssc unregistered [ 9525.979736] LNet: 248773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9526.010674] LNetError: 248773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9526.049227] LNet: Removed LNI 192.168.202.152@tcp [ 9526.877720] Key type .llcrypt unregistered [ 9526.879857] Key type ._llcrypt unregistered [ 9555.339947] Key type ._llcrypt registered [ 9555.346354] Key type .llcrypt registered [ 9555.453508] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9574.143717] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9575.978730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9576.065602] alg: No test for adler32 (adler32-zlib) [ 9577.222917] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 9577.600406] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9579.263652] Key type lgssc registered [ 9580.449107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9633.085946] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9643.743940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9643.762659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.992907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9645.030642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9645.110271] Lustre: lustre-MDT0000: new disk, initializing [ 9645.213404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.232727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9649.883399] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9660.790517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.899855] Lustre: 252842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9660.943488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9660.946243] Lustre: Skipped 1 previous similar message [ 9661.004642] Lustre: lustre-MDT0001: new disk, initializing [ 9661.111440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9661.154294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9661.185435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9666.219299] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9671.504986] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9678.592354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.922327] Lustre: lustre-OST0000: new disk, initializing [ 9678.928114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9678.936833] Lustre: 254474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9679.105942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.827632] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9687.160743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9687.187274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9692.683345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9699.009541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9699.314328] Lustre: lustre-OST0001: new disk, initializing [ 9699.330307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9699.353125] Lustre: 255344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9699.621878] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9707.148957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9707.167208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9707.214549] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9707.348511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9721.510261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9733.338829] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9748.467133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9748.467377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.475385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.504908] Lustre: Skipped 2 previous similar messages [ 9752.861880] Lustre: server umount lustre-MDT0000 complete [ 9757.477199] LustreError: 252832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586586 with bad export cookie 15189403220356543697 [ 9757.488744] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9757.494911] LustreError: 252832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9757.977821] Lustre: server umount lustre-MDT0001 complete [ 9773.461501] Lustre: server umount lustre-OST0000 complete [ 9774.559292] Lustre: 250346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788586587/real 1788586587] req@ffff9cbaf5b77480 x1875468779341952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788586603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9774.620872] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.636213] Lustre: Skipped 1 previous similar message [ 9777.778586] Lustre: server umount lustre-OST0001 complete [ 9792.701842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9803.477521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9813.346308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9823.957996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9845.416863] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9857.178519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.201798] Lustre: lustre-MDT0000: reset Object Index mappings [ 9857.716637] LustreError: 260364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9857.742365] LustreError: 260364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9857.803278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.141831] LustreError: 260365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.148826] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9867.238561] LustreError: 260364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.356251] LustreError: 260365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.951892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.985316] Lustre: lustre-MDT0001: reset Object Index mappings [ 9873.472274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9878.418536] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9881.519565] Lustre: 261506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9887.829231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.331840] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9898.415903] LustreError: 261860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.541539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9903.911690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9904.298921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9904.305854] Lustre: Skipped 1 previous similar message [ 9909.770909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9910.866231] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9918.683770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9921.889817] Lustre: 263376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9928.504578] Lustre: server umount lustre-MDT0000 complete [ 9929.698736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9929.708868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.725900] LustreError: 260365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.748896] LustreError: 260365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9932.170882] LustreError: 261108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586761 with bad export cookie 15189403220356550487 [ 9932.174201] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9932.186896] LustreError: 261108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9932.623219] Lustre: server umount lustre-MDT0001 complete [ 9946.784457] Lustre: server umount lustre-OST0000 complete [ 9960.592413] Lustre: server umount lustre-OST0001 complete [ 9980.966887] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:40:08 (1788586808) [ 9997.845273] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [10009.145452] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10048.177661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.353659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10048.383254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10048.445591] Lustre: lustre-MDT0000: new disk, initializing [10048.520406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10048.543772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10053.828649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10064.315370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.440923] Lustre: 267631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10064.520833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10064.528205] Lustre: Skipped 1 previous similar message [10064.612174] Lustre: lustre-MDT0001: new disk, initializing [10064.780910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10064.790688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10068.904596] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10077.919491] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10084.092785] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10091.210474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10091.485565] Lustre: lustre-OST0000: new disk, initializing [10091.488709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10091.495448] Lustre: 269599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10093.521927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10093.533237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10093.722768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10097.986717] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10108.372442] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10113.921260] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10114.121737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10119.469665] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10119.815425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10124.258112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10124.265530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10124.280634] Lustre: Skipped 3 previous similar messages [10124.289135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10124.291650] Lustre: Skipped 3 previous similar messages [10128.156557] Lustre: server umount lustre-OST0000 complete [10135.008793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.025841] Lustre: Skipped 1 previous similar message [10135.033277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10135.040283] Lustre: Skipped 1 previous similar message [10140.128835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10140.137399] Lustre: Skipped 1 previous similar message [10145.249863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10145.260105] Lustre: Skipped 1 previous similar message [10146.271347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10146.611353] Lustre: server umount lustre-MDT0000 complete [10150.374960] LustreError: 268526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.387439] LustreError: 268526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10150.432902] LustreError: 267623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586979 with bad export cookie 15189403220356552573 [10150.444566] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10150.445983] LustreError: 267623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10150.917834] Lustre: server umount lustre-MDT0001 complete [10169.079406] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10178.635161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10179.096052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10179.104207] Lustre: Skipped 2 previous similar messages [10183.399296] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10184.163365] LustreError: 272537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10184.174663] LustreError: 272537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10193.048502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.979792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10201.243739] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10208.357773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10215.443553] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10225.439737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10225.633169] Lustre: lustre-OST0001: new disk, initializing [10225.636059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10225.642978] Lustre: 274844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10227.780119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10227.785943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10227.955399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10232.999871] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10242.838311] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10247.123147] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10253.373732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10253.476543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10265.061461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10265.073508] LustreError: Skipped 1 previous similar message [10265.082102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.109987] Lustre: Skipped 1 previous similar message [10265.122864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.919532] Lustre: server umount lustre-MDT0000 complete [10274.282600] LustreError: 273266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10274.312562] LustreError: 273266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10275.570032] LustreError: 275610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587104 with bad export cookie 15189403220356553462 [10275.577338] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10275.586623] LustreError: 275610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10276.217584] Lustre: server umount lustre-MDT0001 complete [10289.985268] Lustre: server umount lustre-OST0000 complete [10304.977422] Lustre: server umount lustre-OST0001 complete [10325.975399] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10336.934199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10337.319077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10337.324047] Lustre: Skipped 3 previous similar messages [10342.728719] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10351.698086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.341088] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10361.479948] Lustre: 279335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10367.703950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10376.044780] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10382.561178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10386.444125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.085542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10394.621512] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10402.294065] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10405.346761] Lustre: 281212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10410.856332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10410.870876] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10425.904356] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:47:33 (1788587253) [10443.246449] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10443.248215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10443.256848] Lustre: Skipped 6 previous similar messages [10443.278024] Lustre: Skipped 7 previous similar messages [10447.000587] Lustre: server umount lustre-MDT0000 complete [10448.357889] LustreError: 278195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.388704] LustreError: 278195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10450.340055] LustreError: 278177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587279 with bad export cookie 15189403220356560133 [10450.340975] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10450.346382] LustreError: 278177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10456.780786] Lustre: server umount lustre-MDT0001 complete [10467.112749] Lustre: server umount lustre-OST0000 complete [10471.400524] Lustre: server umount lustre-OST0001 complete [10479.034899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [10489.581977] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10538.535764] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10548.500704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.807190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10548.830513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10548.901465] Lustre: lustre-MDT0000: new disk, initializing [10549.012758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10553.551229] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10563.385655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.534131] Lustre: 285743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10563.793206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10568.287993] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10580.244145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.514330] Lustre: 287376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10582.070267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10588.824634] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10602.736512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10602.935951] Lustre: 288250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10603.023576] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10603.030910] Lustre: Skipped 6 previous similar messages [10604.424778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10604.456830] Lustre: Skipped 1 previous similar message [10604.628939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10609.211529] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10619.280893] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10623.698738] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10623.701282] Lustre: Skipped 1 previous similar message [10638.988436] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:51:06 (1788587466) [10643.589394] Lustre: server umount lustre-MDT0000 complete [10645.481258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10645.505299] Lustre: Skipped 5 previous similar messages [10648.256593] LustreError: 285736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587477 with bad export cookie 15189403220356567847 [10648.266108] LustreError: 285736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10648.266865] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10648.781701] Lustre: server umount lustre-MDT0001 complete [10663.089483] Lustre: server umount lustre-OST0000 complete [10666.463230] Lustre: 250346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587479/real 1788587479] req@ffff9cb9c3afed80 x1875468779656192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788587495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10667.419314] Lustre: server umount lustre-OST0001 complete [10676.807288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.236899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10692.713482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.058874] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10706.110606] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10712.191868] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10719.236195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.666865] LustreError: 292326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.696142] LustreError: 292326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10726.778867] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10734.502143] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10755.009952] loop1: detected capacity change from 0 to 204800000 [10755.292678] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10755.305713] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10755.315432] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10760.926056] loop1: detected capacity change from 0 to 204800000 [10767.023303] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.121474] LustreError: 290484:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs4585: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10767.137486] LustreError: 290484:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs4585 [10767.149102] LustreError: 290484:0:(mgs_handler.c:517:mgs_target_reg()) fs4585-MDT0001: Error checking target index: rc = -13 [10767.182240] LustreError: fs4585-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10767.193102] LustreError: 294218:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10767.199378] LustreError: 294218:0:(tgt_mount.c:2132:server_put_super()) no obd fs4585-MDT0001 [10767.203674] LustreError: 294218:0:(tgt_mount.c:116:server_deregister_mount()) fs4585-MDT0001 not registered [10767.237522] Lustre: server umount fs4585-MDT0001 complete [10767.242375] LustreError: 294218:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10776.785587] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.018669] Lustre: Setting parameter fs4585-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs4585-MDT0001 [10777.106262] Lustre: srv-fs4585-MDT0001: No data found on store. Initialize space. [10777.111859] Lustre: Skipped 4 previous similar messages [10777.272968] Lustre: fs4585-MDT0001: new disk, initializing [10777.282052] Lustre: Skipped 3 previous similar messages [10777.415516] Lustre: srv-fs4585-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10777.426348] Lustre: 294684:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4585-MDT0001-mdtlov: can't create new llog: rc = -115 [10777.452567] LustreError: 294684:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4585-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10783.071776] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10790.544768] Lustre: server umount fs4585-MDT0001 complete [10804.026502] loop1: detected capacity change from 0 to 204800000 [10804.711874] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10809.921530] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10810.355806] Lustre: 296252:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4585-MDT0001-mdtlov: can't create new llog: rc = -115 [10810.380799] LustreError: 296252:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4585-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10816.454597] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10820.090556] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10821.962325] Lustre: server umount fs4585-MDT0001 complete [10835.436894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10835.457996] Lustre: Skipped 1 previous similar message [10835.476732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10835.487551] Lustre: Skipped 6 previous similar messages [10840.504988] Lustre: server umount lustre-MDT0000 complete [10851.086535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.227922] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10856.993840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10857.349501] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10865.654907] loop1: detected capacity change from 0 to 204800000 [10872.090631] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.599542] Lustre: 298699:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4585-MDT0001-mdtlov: can't create new llog: rc = -115 [10872.614658] LustreError: 298699:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4585-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10878.034833] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10883.054092] Lustre: server umount fs4585-MDT0001 complete [10908.140904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10908.152717] Lustre: Skipped 2 previous similar messages [10914.238317] Lustre: server umount lustre-OST0000 complete [10918.390924] LustreError: 250343:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9cb9c388ca80 x1875468779766528/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10923.494704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10924.843931] Lustre: server umount lustre-MDT0000 complete [10928.723466] LustreError: 291239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587757 with bad export cookie 15189403220356570528 [10928.724961] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10928.734088] LustreError: 291239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10929.067556] Lustre: server umount lustre-MDT0001 complete [10942.417914] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [10946.235990] Key type lgssc unregistered [10946.614525] LNet: 301042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10946.624122] LNetError: 301042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10946.654365] LNet: Removed LNI 192.168.202.152@tcp [10947.893167] Key type .llcrypt unregistered [10947.895211] Key type ._llcrypt unregistered [10971.431394] Key type ._llcrypt registered [10971.433913] Key type .llcrypt registered [10971.558437] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10972.878179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10973.114107] alg: No test for adler32 (adler32-zlib) [10974.436232] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [10974.781607] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [10976.415470] Key type lgssc registered [10977.825975] Lustre: Echo OBD driver; http://www.lustre.org/ [10986.364877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10986.410257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10987.968362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10992.802832] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10998.242153] LustreError: 302313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.272204] LustreError: 302313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11001.863726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11002.435856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11006.981743] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11014.520272] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11020.338190] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11027.284369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11027.644353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11033.765175] LustreError: 304141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11033.854772] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11039.213787] LustreError: 304143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.118966] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11044.333222] LustreError: 304492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.919889] LustreError: 304141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.936130] LustreError: 304141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11060.162546] LustreError: 304143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.189565] LustreError: 304143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11066.675893] loop1: detected capacity change from 0 to 204800000 [11066.902183] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11066.914554] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11066.923910] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11071.622377] loop1: detected capacity change from 0 to 204800000 [11076.459354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11076.607253] Lustre: fs587-OST0000: new disk, initializing [11076.611819] Lustre: srv-fs587-OST0000: No data found on store. Initialize space. [11076.618074] Lustre: 306049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11076.692092] Lustre: fs587-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11078.131788] LustreError: 304142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11078.166377] LustreError: 304142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11085.474357] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11095.249676] Lustre: server umount fs587-OST0000 complete [11108.717431] loop1: detected capacity change from 0 to 204800000 [11111.368079] LustreError: 304143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.393163] LustreError: 304143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11113.968467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11115.936261] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11116.149405] Lustre: fs587-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11123.749341] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11129.324855] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11133.861196] loop2: detected capacity change from 0 to 204800000 [11134.223500] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11134.242908] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11134.274332] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11140.263198] loop2: detected capacity change from 0 to 204800000 [11146.438668] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11146.573367] LustreError: fs587-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11146.596475] LustreError: fs587-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11146.614842] LustreError: 309090:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11146.624345] LustreError: 309090:0:(tgt_mount.c:2132:server_put_super()) no obd fs587-OST0001 [11146.630526] LustreError: 309090:0:(tgt_mount.c:116:server_deregister_mount()) fs587-OST0001 not registered [11146.669673] Lustre: server umount fs587-OST0001 complete [11146.678056] LustreError: 309090:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11158.020917] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.141326] Lustre: fs587-OST0001: new disk, initializing [11158.145384] Lustre: srv-fs587-OST0001: No data found on store. Initialize space. [11158.152194] Lustre: 309640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11158.218561] Lustre: fs587-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11165.661983] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11175.251081] Lustre: server umount fs587-OST0000 complete [11175.397268] LustreError: 304141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.416656] LustreError: 304141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [11184.604938] Lustre: server umount fs587-OST0001 complete [11197.310403] loop1: detected capacity change from 0 to 204800000 [11202.422025] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.678424] Lustre: fs587-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11209.936775] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11218.267696] loop2: detected capacity change from 0 to 204800000 [11225.302526] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11233.313238] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11243.252360] Lustre: server umount fs587-OST0000 complete [11253.871865] Lustre: server umount fs587-OST0001 complete [11267.583462] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11269.602067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11269.625078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11269.653547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11272.672323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.677864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11275.068863] Lustre: server umount lustre-OST0000 complete [11278.817533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11278.822891] LustreError: Skipped 1 previous similar message [11278.825581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.841529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11278.845764] Lustre: Skipped 1 previous similar message [11282.913603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11282.915509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11282.936789] Lustre: Skipped 1 previous similar message [11288.034182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.038825] Lustre: Skipped 1 previous similar message [11293.151813] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11293.437745] Lustre: server umount lustre-MDT0000 complete [11297.694087] LustreError: 302294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588126 with bad export cookie 929724494696369498 [11297.706437] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11298.074766] Lustre: server umount lustre-MDT0001 complete [11310.704398] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11313.441645] Key type lgssc unregistered [11313.660136] LNet: 315576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11313.665739] LNetError: 315576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11313.681364] LNet: Removed LNI 192.168.202.152@tcp [11314.797509] Key type .llcrypt unregistered [11314.803387] Key type ._llcrypt unregistered [11356.863557] Key type ._llcrypt registered [11356.867447] Key type .llcrypt registered [11356.966147] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [11376.385500] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11378.321522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11378.506076] alg: No test for adler32 (adler32-zlib) [11379.661476] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11379.934919] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11381.631256] Key type lgssc registered [11383.049819] Lustre: Echo OBD driver; http://www.lustre.org/ [11425.833088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11425.880483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.297244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11427.345412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11427.501239] Lustre: lustre-MDT0000: new disk, initializing [11427.683139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11427.730451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11432.415558] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11443.136208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11443.344796] Lustre: 319436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11443.436537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11443.442722] Lustre: Skipped 1 previous similar message [11443.529368] Lustre: lustre-MDT0001: new disk, initializing [11443.618082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11443.653547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11443.667117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11448.407585] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11458.325506] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11464.949792] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11472.507337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11472.772074] Lustre: lustre-OST0000: new disk, initializing [11472.776996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11472.788543] Lustre: 321407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11472.937073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11478.065244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11478.074718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11478.240936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11480.184401] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11490.488913] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11492.749817] LustreError: 317607:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [11492.799055] Lustre: Mounted lustre-client - version 2.17.58_2_gfa0a981 [11493.764826] Lustre: Unmounted lustre-client [11504.104586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11504.115501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11504.138917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11504.155671] Lustre: Skipped 1 previous similar message [11508.195730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.202957] Lustre: Skipped 1 previous similar message [11509.574056] Lustre: server umount lustre-OST0000 complete [11513.320210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.326701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11513.333188] Lustre: Skipped 2 previous similar messages [11518.434741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.451167] Lustre: Skipped 2 previous similar messages [11519.578644] Lustre: server umount lustre-MDT0000 complete [11523.787354] LustreError: 321417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588352 with bad export cookie 3599070474724160943 [11523.793696] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11523.797811] LustreError: 321417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11524.603878] Lustre: server umount lustre-MDT0001 complete [11537.025909] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11540.716257] Key type lgssc unregistered [11549.219250] LNet: 323566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11549.234450] LNetError: 323566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11621.993795] LNet: Removed LNI 192.168.202.152@tcp [11623.315873] Key type .llcrypt unregistered [11623.317845] Key type ._llcrypt unregistered [11644.285752] Key type ._llcrypt registered [11644.290962] Key type .llcrypt registered [11644.450954] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [11664.796424] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11665.789228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11665.806214] alg: No test for adler32 (adler32-zlib) [11667.081073] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11667.375948] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11669.111234] Key type lgssc registered [11670.258340] Lustre: Echo OBD driver; http://www.lustre.org/ [11704.978778] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:08:51 (1788588531) [11714.850241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11714.947766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.236714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11716.292075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11716.436673] Lustre: lustre-MDT0000: new disk, initializing [11716.571707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11716.608468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11722.602080] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11734.711573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.893548] Lustre: 326783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11734.939294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11734.942638] Lustre: Skipped 1 previous similar message [11735.000528] Lustre: lustre-MDT0001: new disk, initializing [11735.110809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11735.156725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11735.190085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11739.900455] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11750.858619] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11757.356284] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11764.570860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11765.027567] Lustre: lustre-OST0000: new disk, initializing [11765.031336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11765.037554] Lustre: 328754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11765.105448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11771.134877] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11771.999851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11772.006517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11772.153702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11781.122790] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11797.984291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11797.996851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.007211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11801.840826] Lustre: server umount lustre-OST0000 complete [11806.692198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11806.696357] LustreError: Skipped 1 previous similar message [11806.702329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11806.717314] Lustre: Skipped 1 previous similar message [11806.723222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.725442] Lustre: Skipped 1 previous similar message [11808.224764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11808.227609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11808.250575] Lustre: Skipped 1 previous similar message [11811.696885] Lustre: server umount lustre-MDT0000 complete [11815.495298] LustreError: 326774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588644 with bad export cookie 8630458065605069255 [11815.508631] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11815.511574] LustreError: 326774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11816.204965] Lustre: server umount lustre-MDT0001 complete [11830.466030] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11833.621538] Key type lgssc unregistered [11834.098748] LNet: 330662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11834.107177] LNetError: 330662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11834.143809] LNet: Removed LNI 192.168.202.152@tcp [11835.160432] Key type .llcrypt unregistered [11835.164101] Key type ._llcrypt unregistered [11856.580203] Key type ._llcrypt registered [11856.584837] Key type .llcrypt registered [11856.769129] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11858.380778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11858.410044] alg: No test for adler32 (adler32-zlib) [11860.163352] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11860.812213] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11862.519187] Key type lgssc registered [11864.966075] Lustre: Echo OBD driver; http://www.lustre.org/ [11876.137778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11876.156486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11877.816328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11882.280552] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11887.091221] LustreError: 331947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11887.122431] LustreError: 331947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11892.193466] LustreError: 331948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.880940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.396844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11898.247766] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11906.379407] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11912.908937] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11920.140725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11920.550760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11927.808656] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11936.239995] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11947.452630] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11949.089865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11953.633652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11953.648915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11953.649594] Lustre: Skipped 2 previous similar messages [11953.657113] Lustre: Skipped 1 previous similar message [11954.635606] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [11954.642512] Lustre: Skipped 1 previous similar message [11958.753481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11958.768047] Lustre: Skipped 2 previous similar messages [11959.748102] LustreError: 331942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.539928] Lustre: server umount lustre-MDT0000 complete [11963.872709] LustreError: 331947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11963.898639] LustreError: 331947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11966.369649] LustreError: 331928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588795 with bad export cookie 2093728404348670441 [11966.396533] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11966.398579] LustreError: 331928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11968.993886] LustreError: 331944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.994534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.019831] LustreError: 331944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11969.054394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11972.836549] Lustre: server umount lustre-MDT0001 complete [11976.794229] Lustre: server umount lustre-OST0000 complete [11986.320531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.214709] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11995.446938] Lustre: 336128:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.152@tcp [11997.785754] Lustre: server umount MGS complete [12006.877454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12007.506530] LustreError: 336701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.531718] LustreError: 336701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12007.603394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12012.868597] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12022.433470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.756714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12028.046920] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12037.052900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12037.598720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12044.594607] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12052.234434] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12053.999127] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12060.631802] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12061.729406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12067.296960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12067.300515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.307921] LustreError: Skipped 1 previous similar message [12067.310354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12067.310362] Lustre: Skipped 2 previous similar messages [12067.352168] Lustre: Skipped 1 previous similar message [12068.562906] Lustre: server umount lustre-OST0000 complete [12073.952060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12073.954480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.539306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.555993] Lustre: Skipped 3 previous similar messages [12087.263947] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12087.681833] Lustre: server umount lustre-MDT0000 complete [12087.783277] LustreError: 336701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.798966] LustreError: 336701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12091.832461] LustreError: 336683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588921 with bad export cookie 2093728404348671274 [12091.839049] LustreError: 336683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12091.846472] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12092.153404] Lustre: server umount lustre-MDT0001 complete [12105.191344] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12109.107856] Key type lgssc unregistered [12109.515349] LNet: 340136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12109.524196] LNetError: 340136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12109.552223] LNet: Removed LNI 192.168.202.152@tcp [12110.398380] Key type .llcrypt unregistered [12110.404934] Key type ._llcrypt unregistered [12131.327464] Key type ._llcrypt registered [12131.330348] Key type .llcrypt registered [12131.493421] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [12149.572813] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12150.930296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12150.982440] alg: No test for adler32 (adler32-zlib) [12152.291925] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [12152.588848] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [12154.343187] Key type lgssc registered [12155.690636] Lustre: Echo OBD driver; http://www.lustre.org/ [12190.273148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12190.321870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12191.513265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12191.547809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12191.609890] Lustre: lustre-MDT0000: new disk, initializing [12191.732459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12191.756629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12196.881703] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12207.383329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12207.600344] Lustre: 343295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12207.712971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12207.716280] Lustre: Skipped 1 previous similar message [12207.830444] Lustre: lustre-MDT0001: new disk, initializing [12207.993434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12208.057836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12208.086779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12212.590260] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12222.078776] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12227.775983] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12235.665272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12236.054224] Lustre: lustre-OST0000: new disk, initializing [12236.059093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12236.069503] Lustre: 345261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12236.247448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12242.552815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12242.587209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12242.777936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12243.599686] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12251.834535] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12257.759885] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12258.212616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12264.684969] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12265.061100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12268.528710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12268.531236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12268.545727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12268.574661] Lustre: Skipped 1 previous similar message [12273.567085] Lustre: server umount lustre-OST0000 complete [12278.753116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.771514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12278.774661] Lustre: Skipped 2 previous similar messages [12283.872813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12283.888733] Lustre: Skipped 1 previous similar message [12284.332465] Lustre: server umount lustre-MDT0000 complete [12289.085797] LustreError: 343286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589118 with bad export cookie 66350052196008004 [12289.086586] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12289.102884] LustreError: 343286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12289.568567] Lustre: server umount lustre-MDT0001 complete [12299.414663] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:18:46 (1788589126) [12316.662847] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [12327.791324] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12367.035767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12367.288115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12367.353526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12367.488067] Lustre: lustre-MDT0000: new disk, initializing [12367.557938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12367.583684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12372.634212] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12383.948947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.156397] Lustre: 350024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12384.209673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12384.221025] Lustre: Skipped 1 previous similar message [12384.336279] Lustre: lustre-MDT0001: new disk, initializing [12384.527196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12384.601587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12384.625025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12389.751899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12400.523917] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12407.051339] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12414.780473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12415.112833] Lustre: lustre-OST0000: new disk, initializing [12415.116539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12415.121426] Lustre: 351993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12415.212943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12421.652640] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12422.178761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12422.189868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12422.360760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12431.918252] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12468.192747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12468.199994] LustreError: Skipped 1 previous similar message [12468.209901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.218861] Lustre: Skipped 1 previous similar message [12468.227971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12473.823945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12473.829993] Lustre: Skipped 1 previous similar message [12480.991170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12481.203868] Lustre: server umount lustre-OST0000 complete [12487.136391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12487.141550] LustreError: Skipped 1 previous similar message [12487.157282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12487.177796] Lustre: Skipped 1 previous similar message [12487.186704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12487.192203] Lustre: Skipped 3 previous similar messages [12490.803940] Lustre: server umount lustre-MDT0000 complete [12494.309575] LustreError: 350034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12494.335303] LustreError: 350034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12494.984555] LustreError: 350015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589324 with bad export cookie 66350052196008893 [12494.986540] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12495.000103] LustreError: 350015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12495.490950] Lustre: server umount lustre-MDT0001 complete [12509.617983] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12512.861612] Key type lgssc unregistered [12513.245483] LNet: 354168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12513.269433] LNetError: 354168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12513.290351] LNet: Removed LNI 192.168.202.152@tcp [12514.213614] Key type .llcrypt unregistered [12514.216937] Key type ._llcrypt unregistered [12607.085802] Key type ._llcrypt registered [12607.088440] Key type .llcrypt registered [12607.180278] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12607.523069] Key type .llcrypt unregistered [12607.526813] Key type ._llcrypt unregistered [12697.689887] Key type ._llcrypt registered [12697.693582] Key type .llcrypt registered [12697.820504] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12698.206531] Key type .llcrypt unregistered [12698.210413] Key type ._llcrypt unregistered [12732.073278] Key type ._llcrypt registered [12732.078265] Key type .llcrypt registered [12732.209563] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12732.526497] Key type .llcrypt unregistered [12732.530428] Key type ._llcrypt unregistered [12767.258658] Key type ._llcrypt registered [12767.264333] Key type .llcrypt registered [12767.380949] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12767.705652] Key type .llcrypt unregistered [12767.710323] Key type ._llcrypt unregistered [12852.491702] Key type ._llcrypt registered [12852.493833] Key type .llcrypt registered [12852.620729] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12852.940767] Key type .llcrypt unregistered [12852.942918] Key type ._llcrypt unregistered [12886.298773] Key type ._llcrypt registered [12886.300488] Key type .llcrypt registered [12886.383961] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12886.716510] Key type .llcrypt unregistered [12886.719539] Key type ._llcrypt unregistered [12921.551131] Key type ._llcrypt registered [12921.555325] Key type .llcrypt registered [12921.832423] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12922.308322] Key type .llcrypt unregistered [12922.310635] Key type ._llcrypt unregistered [12957.486607] Key type ._llcrypt registered [12957.495212] Key type .llcrypt registered [12957.701603] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12958.080935] Key type .llcrypt unregistered [12958.089099] Key type ._llcrypt unregistered [12990.410954] Key type ._llcrypt registered [12990.413278] Key type .llcrypt registered [12990.620758] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12991.010839] Key type .llcrypt unregistered [12991.015276] Key type ._llcrypt unregistered [13026.230486] Key type ._llcrypt registered [13026.235629] Key type .llcrypt registered [13026.354756] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13026.717290] Key type .llcrypt unregistered [13026.722601] Key type ._llcrypt unregistered [13094.057665] Key type ._llcrypt registered [13094.061312] Key type .llcrypt registered [13094.239877] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13094.680687] Key type .llcrypt unregistered [13094.683858] Key type ._llcrypt unregistered [13128.281495] Key type ._llcrypt registered [13128.284407] Key type .llcrypt registered [13128.417238] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13128.743698] Key type .llcrypt unregistered [13128.746261] Key type ._llcrypt unregistered [13164.005163] Key type ._llcrypt registered [13164.007532] Key type .llcrypt registered [13164.125628] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13164.463338] Key type .llcrypt unregistered [13164.465063] Key type ._llcrypt unregistered [13225.768543] Key type ._llcrypt registered [13225.770834] Key type .llcrypt registered [13225.946580] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [13243.510786] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13244.510355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13244.614231] alg: No test for adler32 (adler32-zlib) [13245.654807] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [13245.882244] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [13247.591211] Key type lgssc registered [13248.955102] Lustre: Echo OBD driver; http://www.lustre.org/ [13284.104217] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13284.162612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13285.370845] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13285.394709] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13285.463820] Lustre: tst250fs-MDT0000: new disk, initializing [13285.541071] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13285.569177] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13289.969408] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13300.168945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13300.318439] Lustre: 374455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13300.398460] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13300.403454] Lustre: Skipped 1 previous similar message [13300.507201] Lustre: tst250fs-MDT0001: new disk, initializing [13300.622610] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13300.659380] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13300.679419] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13305.799110] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13311.281510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13317.885665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13318.274875] Lustre: tst250fs-OST0000: new disk, initializing [13318.282959] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13318.289657] Lustre: 376089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13318.417277] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13325.298869] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13326.414370] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13326.427864] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13326.547548] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13336.128638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13336.324399] Lustre: tst250fs-OST0001: new disk, initializing [13336.327990] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13336.338937] Lustre: 376957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13336.478662] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13342.351693] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13342.370450] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13342.580596] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13343.419159] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13348.996740] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13351.775647] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13351.781326] Lustre: Skipped 2 previous similar messages [13368.296348] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13368.299902] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13368.309089] Lustre: Skipped 2 previous similar messages [13368.312708] Lustre: Skipped 1 previous similar message [13373.420319] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13373.428022] Lustre: Skipped 3 previous similar messages [13374.313291] Lustre: server umount tst250fs-MDT0000 complete [13377.904923] LustreError: 374446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590207 with bad export cookie 11119954956777449682 [13377.907267] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13377.915392] LustreError: 374446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13378.215299] Lustre: server umount tst250fs-MDT0001 complete [13391.754563] Lustre: server umount tst250fs-OST0000 complete [13394.658049] Lustre: 372682:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590207/real 1788590207] req@ffff9cb9cc15f800 x1875472626551808/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788590223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13394.719514] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13394.748434] Lustre: Skipped 1 previous similar message [13395.567916] Lustre: server umount tst250fs-OST0001 complete [13413.649520] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [13421.384858] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13455.788702] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:38:03 (1788590283) [13457.466911] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13475.007221] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [13487.249749] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13525.231598] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13267 sec ========= 02:39:13 (1788590353) [13527.301256] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:39:14 (1788590354) === [13531.544406] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:39:19 (1788590359) === [13576.614234] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13580.280816] Key type lgssc unregistered [13580.544377] LNet: 386101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13580.558120] LNetError: 386101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13580.573915] LNet: Removed LNI 192.168.202.152@tcp [13581.445222] Key type .llcrypt unregistered [13581.453578] Key type ._llcrypt unregistered