[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 563243835 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003282] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.012007] Yama: becoming mindful. [ 0.012928] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.021453] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025787] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] *** VALIDATE tmpfs *** [ 0.029449] *** VALIDATE proc *** [ 0.030209] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.032000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034027] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.038215] debug: unmapping init [mem 0xffffffff89c59000-0xffffffff89c60fff] [ 0.041136] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.043968] ... bit width: 48 [ 0.044015] ... generic registers: 4 [ 0.045014] ... value mask: 0000ffffffffffff [ 0.046016] ... max period: 00007fffffffffff [ 0.047026] ... fixed-purpose events: 3 [ 0.048013] ... event mask: 000000070000000f [ 0.049315] rcu: Hierarchical SRCU implementation. [ 0.051452] smp: Bringing up secondary CPUs ... [ 0.052575] x86: Booting SMP configuration: [ 0.053020] .... node #0, CPUs: #1 #2 #3 [ 0.056077] smp: Brought up 1 node, 4 CPUs [ 0.058015] smpboot: Max logical packages: 1 [ 0.059020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.176041] node 0 deferred pages initialised in 116ms [ 0.181097] devtmpfs: initialized [ 0.182365] x86/mm: Memory block size: 128MB [ 0.186056] gcov: version magic: 0x41383552 [ 0.189202] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.193169] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.195290] pinctrl core: initialized pinctrl subsystem [ 0.197189] [ 0.197794] ************************************************************* [ 0.200015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203014] ** ** [ 0.205012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.208013] ** ** [ 0.210011] ** This means that this kernel is built to expose internal ** [ 0.212016] ** IOMMU data structures, which may compromise security on ** [ 0.215012] ** your system. ** [ 0.217013] ** ** [ 0.219015] ** If you see this message and you are not debugging the ** [ 0.222011] ** kernel, report this immediately to your vendor! ** [ 0.224012] ** ** [ 0.227015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229022] ************************************************************* [ 0.231689] NET: Registered protocol family 16 [ 0.234434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.237058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.239060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.243109] cpuidle: using governor menu [ 0.244871] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.247498] PCI: Using configuration type 1 for base access [ 0.249231] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.260057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.262026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.266164] cryptd: max_cpu_qlen set to 1000 [ 0.269222] ACPI: Added _OSI(Module Device) [ 0.270011] ACPI: Added _OSI(Processor Device) [ 0.271023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.272029] ACPI: Added _OSI(Processor Aggregator Device) [ 0.277290] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.284619] ACPI: Interpreter enabled [ 0.286070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.288013] ACPI: Using IOAPIC for interrupt routing [ 0.289113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300852] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.309102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.314464] acpiphp: Slot [2] registered [ 0.316130] acpiphp: Slot [5] registered [ 0.318135] acpiphp: Slot [6] registered [ 0.319106] acpiphp: Slot [7] registered [ 0.321164] acpiphp: Slot [8] registered [ 0.322172] acpiphp: Slot [9] registered [ 0.324111] acpiphp: Slot [10] registered [ 0.325115] acpiphp: Slot [3] registered [ 0.327190] acpiphp: Slot [4] registered [ 0.329105] acpiphp: Slot [11] registered [ 0.330082] acpiphp: Slot [12] registered [ 0.331096] acpiphp: Slot [13] registered [ 0.333087] acpiphp: Slot [14] registered [ 0.334109] acpiphp: Slot [15] registered [ 0.335129] acpiphp: Slot [16] registered [ 0.337169] acpiphp: Slot [17] registered [ 0.338109] acpiphp: Slot [18] registered [ 0.340340] acpiphp: Slot [19] registered [ 0.341111] acpiphp: Slot [20] registered [ 0.343114] acpiphp: Slot [21] registered [ 0.344101] acpiphp: Slot [22] registered [ 0.345177] acpiphp: Slot [23] registered [ 0.347115] acpiphp: Slot [24] registered [ 0.348109] acpiphp: Slot [25] registered [ 0.350114] acpiphp: Slot [26] registered [ 0.351109] acpiphp: Slot [27] registered [ 0.353208] acpiphp: Slot [28] registered [ 0.354129] acpiphp: Slot [29] registered [ 0.356121] acpiphp: Slot [30] registered [ 0.357116] acpiphp: Slot [31] registered [ 0.359065] PCI host bridge to bus 0000:00 [ 0.360019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.363027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.365023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.367016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.369020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.372039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.374201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.377089] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.380330] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.390913] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396631] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.401018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.404018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406698] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.409892] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.412046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.415780] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.420012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.430016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.436019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441829] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.484159] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.491018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.498019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546825] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.549987] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.554013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.570631] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.588017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.603017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.611080] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.615013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.618018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.627013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.636720] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.638254] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.639305] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.641228] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.643210] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.647037] iommu: Default domain type: Passthrough [ 0.649451] SCSI subsystem initialized [ 0.650148] ACPI: bus type USB registered [ 0.652120] usbcore: registered new interface driver usbfs [ 0.654097] usbcore: registered new interface driver hub [ 0.656103] usbcore: registered new device driver usb [ 0.658200] pps_core: LinuxPPS API ver. 1 registered [ 0.660012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.663060] PTP clock support registered [ 0.666103] EDAC MC: Ver: 3.0.0 [ 0.668637] PCI: Using ACPI for IRQ routing [ 0.671174] NetLabel: Initializing [ 0.673025] NetLabel: domain hash size = 128 [ 0.674010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676088] NetLabel: unlabeled traffic allowed by default [ 0.678333] vgaarb: loaded [ 0.680195] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.691363] clocksource: Switched to clocksource kvm-clock [ 0.802794] VFS: Disk quotas dquot_6.6.0 [ 0.804207] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.806709] *** VALIDATE ramfs *** [ 0.808102] *** VALIDATE hugetlbfs *** [ 0.809732] pnp: PnP ACPI init [ 0.813178] pnp: PnP ACPI: found 6 devices [ 0.831720] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.835342] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.837748] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.840135] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.842680] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.845432] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.848418] NET: Registered protocol family 2 [ 0.850922] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.856140] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859970] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.865707] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.869261] TCP: Hash tables configured (established 65536 bind 65536) [ 0.872240] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.875597] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878628] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881765] NET: Registered protocol family 1 [ 0.884224] RPC: Registered named UNIX socket transport module. [ 0.886192] RPC: Registered udp transport module. [ 0.887983] RPC: Registered tcp transport module. [ 0.889743] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.892164] NET: Registered protocol family 44 [ 0.893872] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.895768] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.897838] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.900248] PCI: CLS 0 bytes, default 64 [ 0.902746] Unpacking initramfs... [ 2.377702] debug: unmapping init [mem 0xffff90b5bcc54000-0xffff90b5bffbffff] [ 2.384496] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.386924] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.391551] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.896850] Initialise system trusted keyrings [ 2.898746] Key type blacklist registered [ 2.900631] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.910951] zbud: loaded [ 2.914324] *** VALIDATE nfs *** [ 2.915739] *** VALIDATE nfs4 *** [ 2.917496] pstore: using deflate compression [ 2.920317] Platform Keyring initialized [ 3.027449] NET: Registered protocol family 38 [ 3.029030] Key type asymmetric registered [ 3.030326] Asymmetric key parser 'x509' registered [ 3.032134] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.035135] io scheduler mq-deadline registered [ 3.036686] io scheduler kyber registered [ 3.037862] io scheduler bfq registered [ 3.039767] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.042957] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.046163] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.049305] ACPI: Power Button [PWRF] [ 3.055138] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.061690] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.072371] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.076860] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.086900] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.112936] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.141576] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.149127] Non-volatile memory driver v1.3 [ 3.150938] Linux agpgart interface v0.103 [ 3.185090] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.187321] vda: detected capacity change from 0 to 74760192 [ 3.201174] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.203931] vdb: detected capacity change from 0 to 1073741824 [ 3.216560] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.219270] vdc: detected capacity change from 0 to 2621440000 [ 3.232436] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.234710] vdd: detected capacity change from 0 to 2621440000 [ 3.247802] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.250366] vde: detected capacity change from 0 to 4294967296 [ 3.265569] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.268820] vdf: detected capacity change from 0 to 4294967296 [ 3.277663] libphy: Fixed MDIO Bus: probed [ 3.286294] usbcore: registered new interface driver usbserial_generic [ 3.288846] usbserial: USB Serial support registered for generic [ 3.291292] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.295456] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.297118] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.299087] mousedev: PS/2 mouse device common for all mice [ 3.303649] rtc_cmos 00:05: RTC can wake from S4 [ 3.305416] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.307979] rtc_cmos 00:05: registered as rtc0 [ 3.309230] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.311097] intel_pstate: CPU model not supported [ 3.313498] hid: raw HID events driver (C) Jiri Kosina [ 3.315212] usbcore: registered new interface driver usbhid [ 3.315418] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317381] usbhid: USB HID core driver [ 3.317521] drop_monitor: Initializing network drop monitor service [ 3.317645] Initializing XFRM netlink socket [ 3.318993] NET: Registered protocol family 10 [ 3.325905] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.326876] Segment Routing with IPv6 [ 3.329862] NET: Registered protocol family 17 [ 3.332097] mpls_gso: MPLS GSO support [ 3.335730] RAS: Correctable Errors collector initialized. [ 3.337300] AVX version of gcm_enc/dec engaged. [ 3.338180] AES CTR mode by8 optimization enabled [ 3.416385] sched_clock: Marking stable (3416353711, 0)->(4278281329, -861927618) [ 3.420490] registered taskstats version 1 [ 3.423425] Loading compiled-in X.509 certificates [ 3.425817] zswap: loaded using pool lzo/zbud [ 3.452437] Key type big_key registered [ 3.464755] Key type encrypted registered [ 3.466484] ima: No TPM chip found, activating TPM-bypass! [ 3.468232] ima: Allocated hash algorithm: sha1 [ 3.469900] ima: No architecture policies found [ 3.471624] evm: Initialising EVM extended attributes: [ 3.473331] evm: security.selinux [ 3.474350] evm: security.ima [ 3.475081] evm: security.capability [ 3.475898] evm: HMAC attrs: 0x1 [ 3.477987] rtc_cmos 00:05: setting system clock to 2026-08-31 05:04:45 UTC (1788152685) [ 3.482836] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.485683] debug: unmapping init [mem 0xffffffff89982000-0xffffffff89c58fff] [ 3.494110] Write protecting the kernel read-only data: 28672k [ 3.496752] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.499548] debug: unmapping init [mem 0xffffffff88914000-0xffffffff889fffff] [ 3.534496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.539695] systemd[1]: Detected virtualization kvm. [ 3.540793] systemd[1]: Detected architecture x86-64. [ 3.541903] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.567240] systemd[1]: No hostname configured. [ 3.568605] systemd[1]: Set hostname to . [ 3.570080] random: systemd: uninitialized urandom read (16 bytes read) [ 3.571836] systemd[1]: Initializing machine ID from random generator. [ 3.677180] random: systemd: uninitialized urandom read (16 bytes read) [ 3.679182] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.683263] random: systemd: uninitialized urandom read (16 bytes read) [ 3.684735] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.687722] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.228769] device-mapper: uevent: version 1.0.3 [ 4.231383] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.962767] virtio_net virtio0 ens2: renamed from eth0 [ 4.970770] random: fast init done [ 5.058767] scsi host0: ata_piix [ 5.093473] scsi host1: ata_piix [ 5.094957] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.096753] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.840129] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.981044] random: crng init done [ 9.982498] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.438946] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.709820] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.944555] SELinux: Disabled at runtime. [ 12.004182] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.011210] systemd[1]: Detected virtualization kvm. [ 12.013116] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.525214] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.528884] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.533391] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.537147] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.539615] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.551732] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.556763] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 12.668125] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.070594] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.458061] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.533925] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.666132] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.692633] EDAC sbridge: Ver: 1.1.2 [ 15.441342] Key type dns_resolver registered [ 15.743364] NFS: Registering the id_resolver key type [ 15.745975] Key type id_resolver registered [ 15.747729] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 51.218527] libcfs: loading out-of-tree module taints kernel. [ 51.329439] Key type ._llcrypt registered [ 51.331205] Key type .llcrypt registered [ 51.463454] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 73.513313] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 75.179687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.190353] alg: No test for adler32 (adler32-zlib) [ 76.601067] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 77.703647] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 79.575201] Key type lgssc registered [ 81.293192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.583025] hrtimer: interrupt took 6888440 ns [ 98.420744] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.518612] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 160.055610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.087592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.517413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.560900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.736470] Lustre: lustre-MDT0000: new disk, initializing [ 161.865970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.890520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.562057] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.086435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.188389] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.225320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.229218] Lustre: Skipped 1 previous similar message [ 179.370154] Lustre: lustre-MDT0001: new disk, initializing [ 179.445475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.485890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.502411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.429412] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.779347] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.480532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.781678] Lustre: lustre-OST0000: new disk, initializing [ 197.788130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.794817] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.862634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.787042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.797744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.897533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.269735] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.289773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.440336] Lustre: lustre-OST0001: new disk, initializing [ 219.442767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.454744] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.509531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.063545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.958514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.980284] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.080831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.470994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.233722] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.176646] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.797412] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 257.987353] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 260.444956] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 263.164637] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 264.655549] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:09:05 EDT 2026 [ 280.557369] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 281.981192] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.440433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 297.448957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.458248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.025526] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.026949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.050502] Lustre: Skipped 2 previous similar messages [ 301.083824] Lustre: Skipped 3 previous similar messages [ 303.742887] Lustre: server umount lustre-MDT0000 complete [ 307.417122] LustreError: 7479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152989 with bad export cookie 13152586079051902986 [ 307.435080] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.263791] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.265773] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.270680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.294435] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 311.301624] Lustre: Skipped 1 previous similar message [ 312.288816] LustreError: 10519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.316607] LustreError: 10519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 313.732719] Lustre: server umount lustre-MDT0001 complete [ 317.675927] Lustre: server umount lustre-OST0000 complete [ 321.202988] Lustre: server umount lustre-OST0001 complete [ 327.176714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 334.870441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 367.477504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.723798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.751904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.808364] Lustre: lustre-MDT0000: new disk, initializing [ 367.863678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.875257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 372.064971] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 383.330274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.616356] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 383.716200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 383.719528] Lustre: Skipped 1 previous similar message [ 383.809670] Lustre: lustre-MDT0001: new disk, initializing [ 383.960443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 384.027777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 384.044594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.151525] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 399.140782] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 405.781882] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 413.929733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.215684] Lustre: lustre-OST0000: new disk, initializing [ 414.219285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.226374] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.310888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.127838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.136349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.285536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.208911] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 430.826838] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.082979] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.305948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 441.589237] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 441.863282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.948451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.964801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.976120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.981996] Lustre: Skipped 4 previous similar messages [ 452.068445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.207894] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 458.339534] Lustre: server umount lustre-OST0000 complete [ 462.306264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.319569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 462.319578] Lustre: Skipped 3 previous similar messages [ 462.330456] Lustre: Skipped 2 previous similar messages [ 468.359845] Lustre: server umount lustre-MDT0000 complete [ 472.548246] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.572861] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 472.737508] LustreError: 17048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153154 with bad export cookie 13152586079051904519 [ 472.753684] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.757510] LustreError: 17048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 473.290835] Lustre: server umount lustre-MDT0001 complete [ 481.583531] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:12:41 (1788153161) [ 500.179380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.629121] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.695499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.831076] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.424493] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 516.029675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.067975] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.081399] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 520.822865] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 528.571892] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 534.596721] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.912737] Lustre: Failing over lustre-MDT0000 [ 536.931297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.951645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.959797] Lustre: Skipped 3 previous similar messages [ 537.124990] Lustre: server umount lustre-MDT0000 complete [ 541.091575] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.125081] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 557.044672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.302039] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.890365] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 561.896831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 561.929268] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 563.046475] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 573.329360] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 578.180953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.196432] Lustre: Skipped 1 previous similar message [ 578.210810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.217713] Lustre: Skipped 2 previous similar messages [ 584.708922] Lustre: server umount lustre-MDT0000 complete [ 586.733634] LustreError: 22586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.767497] LustreError: 22586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 589.277530] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153271 with bad export cookie 13152586079051905933 [ 589.281317] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.306201] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 590.134961] Lustre: server umount lustre-MDT0001 complete [ 601.478872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.349104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.359957] Lustre: Skipped 2 previous similar messages [ 607.819979] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 617.532857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.190259] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 633.329740] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 640.575573] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 643.030311] Lustre: Failing over lustre-MDT0000 [ 643.554270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.558931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.582919] Lustre: Skipped 1 previous similar message [ 645.262079] Lustre: server umount lustre-MDT0000 complete [ 652.768800] LustreError: 23264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.799941] LustreError: 23264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 663.007181] Lustre: 3656:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153329/real 1788153329] req@ffff90b60727ed80 x1875014067609088/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788153345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 663.046911] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.076742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.248830] LustreError: 3653:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff90b627b04e00 x1875014067611520/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 673.645655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 678.886820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 678.897993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 678.914557] Lustre: Skipped 1 previous similar message [ 678.945284] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 679.094366] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 689.578762] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 694.241339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.253095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.260617] Lustre: Skipped 2 previous similar messages [ 708.066079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 708.201260] Lustre: server umount lustre-MDT0000 complete [ 712.019151] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153394 with bad export cookie 13152586079051906871 [ 712.032741] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.033340] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 712.430697] Lustre: server umount lustre-MDT0001 complete [ 720.090627] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:16:40 (1788153400) [ 738.747806] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 749.239951] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 787.505588] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 798.660690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.901687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 798.930528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 799.061471] Lustre: lustre-MDT0000: new disk, initializing [ 799.172534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.176740] Lustre: Skipped 2 previous similar messages [ 799.204854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 804.857228] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 815.949511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.028170] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 816.257449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 820.592617] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 829.666645] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 835.463907] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 842.390297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.830468] Lustre: 31613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 844.605723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 844.691681] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 844.741366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 844.763292] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 849.443524] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 859.583237] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 865.413753] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 885.739759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.741855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.762221] LustreError: Skipped 1 previous similar message [ 885.763060] Lustre: Skipped 3 previous similar messages [ 885.772326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.779412] Lustre: Skipped 7 previous similar messages [ 889.801681] Lustre: server umount lustre-OST0000 complete [ 900.601106] Lustre: server umount lustre-MDT0000 complete [ 901.095145] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.117526] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 904.193089] LustreError: 30436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153586 with bad export cookie 13152586079051907277 [ 904.204144] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.211295] LustreError: 30436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 904.736393] Lustre: server umount lustre-MDT0001 complete [ 916.964800] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 920.314966] Key type lgssc unregistered [ 920.633174] LNet: 33592:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.639118] LNetError: 33592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.707562] LNet: Removed LNI 192.168.202.102@tcp [ 922.721398] Key type .llcrypt unregistered [ 922.727760] Key type ._llcrypt unregistered [ 948.317197] Key type ._llcrypt registered [ 948.322389] Key type .llcrypt registered [ 948.410405] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 949.385988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.465647] alg: No test for adler32 (adler32-zlib) [ 950.519467] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 950.746732] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 952.399173] Key type lgssc registered [ 953.429859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.065643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.087522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.499959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.730606] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 971.749346] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.769944] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 974.739813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.111663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.473571] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 986.365031] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.022221] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 997.580805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.980116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.066120] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1004.600076] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1006.208699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1012.074335] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1026.441254] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:21:47 (1788153707) [ 1028.506559] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1030.462577] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1037.123264] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:21:57 (1788153717) [ 1046.099882] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:22:06 (1788153726) [ 1047.101863] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1058.011381] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1058.030794] Lustre: Skipped 2 previous similar messages [ 1060.379224] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:22:20 (1788153740) [ 1062.919528] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1062.925174] Lustre: Skipped 2 previous similar messages [ 1081.568341] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1081.574122] Lustre: Skipped 7 previous similar messages [ 1084.711474] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:22:45 (1788153765) [ 1087.045465] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1090.565538] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1099.774080] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:23:00 (1788153780) [ 1102.018391] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1102.024992] Lustre: Skipped 3 previous similar messages [ 1114.549087] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:23:14 (1788153794) [ 1128.416173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.426851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.443954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.442115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.443778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.562471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.573384] Lustre: Skipped 2 previous similar messages [ 1139.680812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.684245] Lustre: Skipped 1 previous similar message [ 1140.191137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1140.351601] Lustre: server umount lustre-OST0000 complete [ 1144.293484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1144.297344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.320985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.335562] Lustre: server umount lustre-MDT0000 complete [ 1153.674336] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153835 with bad export cookie 6544926348781719377 [ 1153.684503] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.134869] Lustre: server umount lustre-MDT0001 complete [ 1163.973288] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1166.038806] Key type lgssc unregistered [ 1166.277171] LNet: 42867:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.288644] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.319446] LNet: Removed LNI 192.168.202.102@tcp [ 1167.167128] Key type .llcrypt unregistered [ 1167.169650] Key type ._llcrypt unregistered [ 1183.155412] Key type ._llcrypt registered [ 1183.157613] Key type .llcrypt registered [ 1183.290748] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 1199.343352] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1200.707750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.743330] alg: No test for adler32 (adler32-zlib) [ 1201.956899] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1202.337349] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1204.055176] Key type lgssc registered [ 1205.556879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.197779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.220794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.558796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1241.586112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1241.691398] Lustre: lustre-MDT0000: new disk, initializing [ 1241.767603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.832197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1246.016418] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1257.301939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.517646] Lustre: 46019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1257.586892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1257.590276] Lustre: Skipped 1 previous similar message [ 1257.658332] Lustre: lustre-MDT0001: new disk, initializing [ 1257.729817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.768866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1257.783728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1262.838541] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1272.402776] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1277.907216] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1284.731109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1285.052554] Lustre: lustre-OST0000: new disk, initializing [ 1285.056770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1285.063426] Lustre: 47988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1285.204128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.176707] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1294.482509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1294.504729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1294.623398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1300.479261] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1305.623547] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1305.934929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1311.333747] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1311.624176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1315.296212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.311489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.312054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.328681] LustreError: Skipped 1 previous similar message [ 1319.973537] Lustre: server umount lustre-OST0000 complete [ 1325.540058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.568514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.574851] Lustre: Skipped 2 previous similar messages [ 1325.598303] Lustre: Skipped 2 previous similar messages [ 1330.657920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.662391] Lustre: Skipped 1 previous similar message [ 1331.640938] Lustre: server umount lustre-MDT0000 complete [ 1335.851296] LustreError: 46010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154017 with bad export cookie 10797288908729217876 [ 1335.861367] LustreError: 46010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1335.861661] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.253935] Lustre: server umount lustre-MDT0001 complete [ 1347.680631] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:27:07 (1788154027) [ 1354.500361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.937825] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.955894] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1355.005889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.429278] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1360.355572] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.458427] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.047372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.466019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.997404] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1382.563918] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1390.042611] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.026242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.375387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.458525] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1415.650590] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1421.754985] LustreError: 52983:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1422.555452] Lustre: Modifying parameter general.timeout=1 in log params [ 1423.655675] Lustre: Modifying parameter general.timeout=2 in log params [ 1426.241665] Lustre: Modifying parameter general.timeout=5 in log params [ 1426.245254] Lustre: Skipped 2 previous similar messages [ 1431.261467] Lustre: Modifying parameter general.timeout=10 in log params [ 1431.266237] Lustre: Skipped 4 previous similar messages [ 1439.996404] Lustre: Modifying parameter general.timeout=19 in log params [ 1439.999522] Lustre: Skipped 8 previous similar messages [ 1456.483276] Lustre: Modifying parameter general.timeout=37 in log params [ 1456.491503] Lustre: Skipped 17 previous similar messages [ 1488.685187] Lustre: Modifying parameter general.timeout=65 in log params [ 1488.691528] Lustre: Skipped 27 previous similar messages [ 1558.452582] Lustre: Disabling parameter general.timeout= in log params [ 1558.455123] Lustre: Skipped 64 previous similar messages [ 1562.191380] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:30:42 (1788154242) [ 1565.185397] Lustre: 59939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1567.399124] Lustre: 60038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1571.004283] Lustre: 60235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1571.016987] Lustre: 60235:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1575.084684] Lustre: 60451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1575.100451] Lustre: 60451:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1602.460823] Lustre: 61805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1602.483253] LustreError: 61805:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1606.814651] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:31:27 (1788154287) [ 1619.424765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1619.430774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.446919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.021358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.039444] Lustre: Skipped 1 previous similar message [ 1625.668435] Lustre: server umount lustre-MDT0000 complete [ 1628.131376] LustreError: 52946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.159025] LustreError: 52946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1629.494169] LustreError: 50433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154311 with bad export cookie 10797288908729218758 [ 1629.507898] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.511087] LustreError: 50433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1629.917863] Lustre: server umount lustre-MDT0001 complete [ 1643.411945] Lustre: server umount lustre-OST0000 complete [ 1671.174293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.610934] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1672.025668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.199667] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1687.074321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.182617] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1687.234896] Lustre: 64335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1692.392416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1700.446116] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1706.116747] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1713.789858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1714.176256] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1714.551524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.556772] Lustre: Skipped 1 previous similar message [ 1721.309823] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1730.147525] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1732.163722] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1732.177326] Lustre: Skipped 23 previous similar messages [ 1732.240243] Lustre: 66902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1742.821399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1742.822092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.854274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1742.864453] Lustre: Skipped 3 previous similar messages [ 1748.176878] Lustre: server umount lustre-OST0000 complete [ 1753.062865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.085184] Lustre: Skipped 1 previous similar message [ 1753.092910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.104606] Lustre: Skipped 4 previous similar messages [ 1758.493412] Lustre: server umount lustre-MDT0000 complete [ 1763.237895] LustreError: 64326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154445 with bad export cookie 10797288908729221887 [ 1763.253631] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.258500] LustreError: 64326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1763.582111] Lustre: server umount lustre-MDT0001 complete [ 1775.763627] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1778.110518] Key type lgssc unregistered [ 1778.361717] LNet: 68111:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.368523] LNetError: 68111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.385034] LNet: Removed LNI 192.168.202.102@tcp [ 1779.593643] Key type .llcrypt unregistered [ 1779.596404] Key type ._llcrypt unregistered [ 1802.041632] Key type ._llcrypt registered [ 1802.054354] Key type .llcrypt registered [ 1802.202103] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1803.273638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1803.459928] alg: No test for adler32 (adler32-zlib) [ 1804.526222] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1804.794321] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1806.511194] Key type lgssc registered [ 1807.646563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.794487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.816744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.470820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.101719] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1826.789422] LustreError: 69519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.802748] LustreError: 69519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1831.283373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.183753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1838.094619] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1847.050851] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1854.624754] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1862.233809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.941397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.604538] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1878.607070] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1897.844720] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1907.170310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.172043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.178459] Lustre: Skipped 2 previous similar messages [ 1907.196982] Lustre: Skipped 2 previous similar messages [ 1912.291245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.295127] Lustre: Skipped 2 previous similar messages [ 1912.782674] Lustre: server umount lustre-MDT0000 complete [ 1916.152317] LustreError: 69501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154598 with bad export cookie 3814403780535295312 [ 1916.166442] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.170633] LustreError: 69501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1916.510884] Lustre: server umount lustre-MDT0001 complete [ 1930.786326] Lustre: server umount lustre-OST0000 complete [ 1962.324108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.521234] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1962.544308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1962.809202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.659849] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1976.345370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.490433] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1976.534798] Lustre: 74410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1976.939489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1981.524533] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1988.354566] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1993.379135] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2000.598926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.776555] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2001.067114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.548838] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2014.426953] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2016.145354] Lustre: 76986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2017.689333] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2026.978342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.980630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.982885] LustreError: Skipped 1 previous similar message [ 2026.991968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.995935] Lustre: Skipped 1 previous similar message [ 2032.097624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.102077] Lustre: Skipped 2 previous similar messages [ 2037.218976] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2037.219859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.237315] Lustre: Skipped 1 previous similar message [ 2037.460365] Lustre: server umount lustre-OST0000 complete [ 2042.337619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.350327] Lustre: Skipped 1 previous similar message [ 2047.093442] Lustre: server umount lustre-MDT0000 complete [ 2047.456446] LustreError: 77021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.462178] LustreError: 77021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2050.870434] LustreError: 74403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154732 with bad export cookie 3814403780535296145 [ 2050.879449] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.885917] LustreError: 74403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2051.247251] Lustre: server umount lustre-MDT0001 complete [ 2063.035433] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2065.980905] Key type lgssc unregistered [ 2066.374901] LNet: 78199:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.387557] LNetError: 78199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.413559] LNet: Removed LNI 192.168.202.102@tcp [ 2067.515867] Key type .llcrypt unregistered [ 2067.517426] Key type ._llcrypt unregistered [ 2093.493481] Key type ._llcrypt registered [ 2093.499281] Key type .llcrypt registered [ 2093.660185] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2095.344575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.519422] alg: No test for adler32 (adler32-zlib) [ 2096.585150] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2096.881371] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2098.631170] Key type lgssc registered [ 2099.735214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.427481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2107.475730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.954493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.035797] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2118.120831] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.150678] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2123.233394] LustreError: 79616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.792031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.472152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2130.616812] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2139.133434] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2145.026241] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2152.541783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.350422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.732539] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2166.571971] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2175.598989] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.558993] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.241843] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.245418] Lustre: Skipped 1 previous similar message [ 2180.853018] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2180.855770] Lustre: Skipped 2 previous similar messages [ 2184.962210] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2184.965282] Lustre: Skipped 4 previous similar messages [ 2193.258530] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2193.263572] Lustre: Skipped 9 previous similar messages [ 2209.903695] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2209.909699] Lustre: Skipped 18 previous similar messages [ 2241.995181] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2242.003447] Lustre: Skipped 34 previous similar messages [ 2273.613540] Lustre: 87714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2276.597639] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:42:36 (1788154956) [ 2278.436404] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2279.944916] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:42:40 (1788154960) [ 2287.597061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2287.602224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.612365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.613879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.621691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.624693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.677477] Lustre: Skipped 1 previous similar message [ 2293.729568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2293.749711] Lustre: Skipped 1 previous similar message [ 2298.335691] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2298.694892] Lustre: server umount lustre-OST0000 complete [ 2304.484431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2304.498838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.537521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.089146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.090096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.550651] Lustre: server umount lustre-MDT0000 complete [ 2314.208735] LustreError: 79611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.217306] LustreError: 79611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2314.681992] LustreError: 84629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154996 with bad export cookie 9750277659116944030 [ 2314.683765] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.690779] LustreError: 84629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2315.087358] Lustre: server umount lustre-MDT0001 complete [ 2330.332893] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2333.541251] Key type lgssc unregistered [ 2333.807933] LNet: 88967:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.812734] LNetError: 88967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.822761] LNet: Removed LNI 192.168.202.102@tcp [ 2334.521395] Key type .llcrypt unregistered [ 2334.522929] Key type ._llcrypt unregistered [ 2339.015672] Key type ._llcrypt registered [ 2339.017616] Key type .llcrypt registered [ 2339.100647] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2343.714367] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules [ 2345.161873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.216697] alg: No test for adler32 (adler32-zlib) [ 2346.186247] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2346.396181] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2348.055278] Key type lgssc registered [ 2349.417805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2350.438350] LustreError: 90169:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2367.669656] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2372.565202] LustreError: 90169:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2378.004853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2378.039493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.688681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.505729] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2389.993451] LustreError: 91194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.014596] LustreError: 91194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2394.118959] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:44:34 (1788155074) [ 2395.109181] LustreError: 91193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.374456] LustreError: 91225:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2399.394200] LustreError: 91225:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2399.407489] LustreError: 91225:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2399.852182] Lustre: server umount lustre-MDT0000 complete [ 2414.683097] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2417.669203] Key type lgssc unregistered [ 2417.978958] LNet: 92718:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.993040] LNetError: 92718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.048301] LNet: Removed LNI 192.168.202.102@tcp [ 2420.025274] Key type .llcrypt unregistered [ 2420.030688] Key type ._llcrypt unregistered [ 2439.966098] Key type ._llcrypt registered [ 2439.967745] Key type .llcrypt registered [ 2440.112595] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2440.973371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.016388] alg: No test for adler32 (adler32-zlib) [ 2442.147502] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2442.382649] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2444.079184] Key type lgssc registered [ 2446.020956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2454.884936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2454.903888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.330449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.457411] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2466.789323] LustreError: 93936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.820776] LustreError: 93936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2468.265319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.895143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2473.375917] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2481.535225] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2487.006951] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2494.172746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.703669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.602862] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2509.650761] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2563.076649] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:47:23 (1788155243) [ 2564.844367] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2567.656687] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:47:27 (1788155247) [ 2574.819993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.825966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.849633] Lustre: Skipped 2 previous similar messages [ 2574.870574] Lustre: Skipped 2 previous similar messages [ 2578.552558] Lustre: server umount lustre-MDT0000 complete [ 2579.941139] LustreError: 96777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.976281] LustreError: 96777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2582.567786] LustreError: 93916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155264 with bad export cookie 2054429885482309386 [ 2582.588603] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.885978] Lustre: server umount lustre-MDT0001 complete [ 2596.959201] Lustre: server umount lustre-OST0000 complete [ 2610.168710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.681083] LustreError: 98212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.940911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.290120] LustreError: 98213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.615852] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2620.386819] LustreError: 98212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.374369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.991693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.259366] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2639.934835] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.310503] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2658.863758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.146401] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2659.165779] LustreError: 98201:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2659.199462] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2659.207652] LustreError: 100296:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2659.216086] LustreError: 100296:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2659.219476] LustreError: 100296:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2659.268913] Lustre: server umount lustre-OST0000 complete [ 2659.272655] LustreError: 100296:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2670.497963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.766214] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2670.774619] LustreError: 98200:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2670.802666] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2670.809962] LustreError: 100761:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2670.814688] LustreError: 100761:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2670.821599] LustreError: 100761:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2670.962786] Lustre: server umount lustre-OST0000 complete [ 2670.979304] LustreError: 100761:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2684.272129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2684.511083] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2684.523675] Lustre: Found index 0 for lustre-OST0000, updating log [ 2684.529893] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2684.625386] Lustre: lustre-OST0000: new disk, initializing [ 2684.628914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2684.642727] Lustre: 101296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2685.050132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.848429] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2697.897540] Lustre: Failing over lustre-OST0000 [ 2698.115573] Lustre: server umount lustre-OST0000 complete [ 2701.404500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.406376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.417726] Lustre: Skipped 1 previous similar message [ 2701.431631] Lustre: Skipped 1 previous similar message [ 2706.915857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2706.920727] Lustre: Skipped 1 previous similar message [ 2707.691964] Lustre: server umount lustre-MDT0000 complete [ 2711.606083] LustreError: 98194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155393 with bad export cookie 2054429885482312130 [ 2711.607364] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.624863] LustreError: 98194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2721.493134] Lustre: server umount lustre-MDT0001 complete [ 2729.908505] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:50:10 (1788155410) [ 2746.260693] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2757.706908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.218960] LustreError: 103646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.243601] LustreError: 103646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2758.339705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.541874] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2769.869614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.768393] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2777.660830] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2783.856930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.854402] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2791.634443] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2795.117110] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2795.170183] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2795.172337] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2801.719146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.872718] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2802.003769] Lustre: lustre-OST0001: new disk, initializing [ 2802.008809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2802.016795] Lustre: 105964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2802.416597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.423670] Lustre: Skipped 2 previous similar messages [ 2809.081938] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2812.534887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2812.545724] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2812.643627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2816.882082] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2823.411214] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2828.269958] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.275652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2828.284302] Lustre: Skipped 1 previous similar message [ 2832.520247] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 2832.537855] Lustre: Skipped 2 previous similar messages [ 2834.258692] Lustre: server umount lustre-MDT0001 complete [ 2837.585590] LustreError: 103641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.599241] LustreError: 103641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2846.141309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.223171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2846.227959] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2846.268601] Lustre: 103634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2847.572862] LustreError: 107496:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2847.579505] LustreError: 107496:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2847.612168] LustreError: 103638:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2847.710747] LustreError: 107505:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -17 [ 2847.716707] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2847.716707] [ 2850.500121] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2852.386595] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2861.580921] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:52:21 (1788155541) [ 2879.870823] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2898.187922] Lustre: 109576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2921.311386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2924.701617] Lustre: 110712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3005.410704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3005.421349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.436818] Lustre: Skipped 1 previous similar message [ 3005.446535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.449199] Lustre: Skipped 3 previous similar messages [ 3011.390211] Lustre: server umount lustre-MDT0000 complete [ 3015.300203] LustreError: 103627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155697 with bad export cookie 2054429885482313131 [ 3015.306290] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3015.322095] LustreError: 103627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3016.003310] Lustre: server umount lustre-MDT0001 complete [ 3030.495478] Lustre: server umount lustre-OST0000 complete [ 3032.544177] Lustre: 93250:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155698/real 1788155698] req@ffff90b604ad5500 x1875016547661568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3032.582521] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.601339] Lustre: Skipped 3 previous similar messages [ 3035.277526] Lustre: server umount lustre-OST0001 complete [ 3042.304767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3055.656045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3064.869676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3076.051785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3097.835360] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3110.061893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.081630] Lustre: lustre-MDT0000: reset Object Index mappings [ 3110.612680] LustreError: 114662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.641256] LustreError: 114662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3110.758218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.779056] Lustre: Skipped 1 previous similar message [ 3115.447510] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3125.227652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.249444] Lustre: lustre-MDT0001: reset Object Index mappings [ 3131.072803] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3134.302283] Lustre: 115810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3142.385669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3150.655392] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3150.915239] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3155.777797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3160.818167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.736243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3166.751636] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3169.934779] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3178.810488] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3183.223450] Lustre: 117686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3231.313131] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:58:32 (1788155912) [ 3258.859866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.863689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3258.884975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3258.897317] Lustre: Skipped 10 previous similar messages [ 3263.902988] Lustre: server umount lustre-MDT0000 complete [ 3263.973222] LustreError: 114658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.995251] LustreError: 114658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3267.440892] LustreError: 114642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155949 with bad export cookie 2054429885482360003 [ 3267.449585] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3267.464481] LustreError: 114642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3273.939425] Lustre: server umount lustre-MDT0001 complete [ 3284.437831] Lustre: server umount lustre-OST0000 complete [ 3302.367845] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3302.609412] Lustre: server umount lustre-OST0001 complete [ 3309.633696] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3323.062559] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3362.252070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.686300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3362.772567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3362.876185] Lustre: lustre-MDT0000: new disk, initializing [ 3363.008746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.014891] Lustre: Skipped 3 previous similar messages [ 3363.066832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3368.508840] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3376.734210] Lustre: server umount lustre-MDT0000 complete [ 3388.307741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.632794] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3388.643730] Lustre: 121973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3388.657549] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3394.305512] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3400.984829] Lustre: server umount lustre-MDT0000 complete [ 3411.765970] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:01:32 (1788156092) [ 3413.420657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3415.857787] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:01:36 (1788156096) [ 3437.901790] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3447.342745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.341581] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3461.371360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.529754] Lustre: 124025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3461.612113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3461.615449] Lustre: Skipped 1 previous similar message [ 3461.772815] Lustre: lustre-MDT0001: new disk, initializing [ 3461.915431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3461.929409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3466.300488] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3470.764430] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3476.911289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.206520] Lustre: lustre-OST0000: new disk, initializing [ 3477.213637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3477.224458] Lustre: 125602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3478.647916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3478.659815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3478.780618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3484.021622] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3496.896765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3497.174797] Lustre: lustre-OST0001: new disk, initializing [ 3497.186295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3497.190867] Lustre: 126472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3499.232615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3499.241408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3499.343845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3503.539901] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3514.425574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3518.586512] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3902.703722] Lustre: 132883:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3902.718641] Lustre: 132883:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3902.726704] Lustre: 132883:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3902.736917] Lustre: 132883:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3902.758743] Lustre: 132883:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3902.772080] Lustre: 132883:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4177.039527] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:14:17 (1788156857) [ 4194.273409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4194.286763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.299493] Lustre: Skipped 5 previous similar messages [ 4194.310886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.318136] Lustre: Skipped 4 previous similar messages [ 4196.323108] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.509586] Lustre: server umount lustre-MDT0000 complete [ 4201.449349] LustreError: 127275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.501225] LustreError: 127275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4203.177155] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156885 with bad export cookie 2054429885482426678 [ 4203.180366] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4203.192991] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4206.585518] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.609650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4206.614016] Lustre: Skipped 3 previous similar messages [ 4206.623551] Lustre: Skipped 5 previous similar messages [ 4209.900072] Lustre: server umount lustre-MDT0001 complete [ 4220.932960] Lustre: server umount lustre-OST0000 complete [ 4230.998247] Lustre: server umount lustre-OST0001 complete [ 4240.135234] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4250.830825] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4292.789165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.288623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4293.358576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4293.491970] Lustre: lustre-MDT0000: new disk, initializing [ 4293.638437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.667622] Lustre: Skipped 5 previous similar messages [ 4293.754425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4301.062306] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4316.171959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.393418] Lustre: 135934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4316.492595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4316.504723] Lustre: Skipped 1 previous similar message [ 4316.633852] Lustre: lustre-MDT0001: new disk, initializing [ 4316.866414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4316.889917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4324.144485] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4334.155387] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4339.953470] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4346.847513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.128995] Lustre: lustre-OST0000: new disk, initializing [ 4347.132795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4347.148735] Lustre: 137913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4347.274977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.279549] Lustre: Skipped 1 previous similar message [ 4348.902065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4348.912218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4349.010066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4355.323778] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4366.406369] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4377.199421] Lustre: lustre-MDD0000: changelog on [ 4380.163978] Lustre: lustre-MDD0001: changelog on [ 6100.718803] Lustre: lustre-MDD0001: changelog off [ 6102.872690] Lustre: 135941:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6104.053565] Lustre: lustre-MDD0000: changelog off [ 6115.296667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6115.302443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.303078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6115.319859] LustreError: Skipped 1 previous similar message [ 6115.328310] Lustre: Skipped 2 previous similar messages [ 6116.857951] Lustre: server umount lustre-OST0000 complete [ 6124.006063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6124.017219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6124.033367] Lustre: Skipped 1 previous similar message [ 6124.047701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6124.052746] Lustre: Skipped 1 previous similar message [ 6130.659445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.669060] Lustre: Skipped 3 previous similar messages [ 6135.263621] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6135.717289] Lustre: server umount lustre-MDT0000 complete [ 6135.785481] LustreError: 135945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.802250] LustreError: 135945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6140.228062] LustreError: 135928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158822 with bad export cookie 2054429885483246077 [ 6140.234594] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6140.239151] LustreError: 135928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6140.625864] Lustre: server umount lustre-MDT0001 complete [ 6151.014986] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6153.917334] Key type lgssc unregistered [ 6154.164881] LNet: 147022:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6154.176287] LNetError: 147022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6154.192655] LNet: Removed LNI 192.168.202.102@tcp [ 6155.033168] Key type .llcrypt unregistered [ 6155.034887] Key type ._llcrypt unregistered [ 6175.144507] Key type ._llcrypt registered [ 6175.149976] Key type .llcrypt registered [ 6175.278245] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6191.964619] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6193.457634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.531917] alg: No test for adler32 (adler32-zlib) [ 6194.712209] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6195.081679] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6196.882832] Key type lgssc registered [ 6198.194758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6240.105317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6240.162127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.889266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6241.966327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6242.123069] Lustre: lustre-MDT0000: new disk, initializing [ 6242.357636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.435504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6247.928154] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6259.312298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6259.484933] Lustre: 150234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6259.521955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6259.529284] Lustre: Skipped 1 previous similar message [ 6259.705714] Lustre: lustre-MDT0001: new disk, initializing [ 6259.855365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.935223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6259.942184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6265.021837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6274.528519] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6280.034389] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6287.452802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.702376] Lustre: lustre-OST0000: new disk, initializing [ 6287.706350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6287.711614] Lustre: 152203:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6287.810152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.069451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6293.078150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6293.198308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6294.746984] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6304.860623] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6311.068708] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_module obdecho/obdecho [ 6312.753044] LustreError: 153206:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6312.762977] LustreError: 153206:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6314.672227] LustreError: 153302:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6326.243308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6326.251415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.273074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.312937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.315200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.328098] Lustre: Skipped 1 previous similar message [ 6329.359304] Lustre: Skipped 1 previous similar message [ 6331.009309] Lustre: server umount lustre-MDT0000 complete [ 6334.415292] LustreError: 150225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159016 with bad export cookie 11342642018544762305 [ 6334.417185] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.432079] LustreError: 150225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6334.473787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.499391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6334.514319] Lustre: Skipped 1 previous similar message [ 6335.044327] Lustre: server umount lustre-MDT0001 complete [ 6340.704254] Lustre: server umount lustre-OST0000 complete [ 6361.473861] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6371.915640] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6408.811735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.027425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6409.056379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6409.129382] Lustre: lustre-MDT0000: new disk, initializing [ 6409.183479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6409.203658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6414.452092] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6426.105755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.314333] Lustre: 157015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6426.474390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6426.484155] Lustre: Skipped 1 previous similar message [ 6426.727968] Lustre: lustre-MDT0001: new disk, initializing [ 6426.934175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6427.014200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6427.032214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6433.789421] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6443.403966] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6448.779185] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6455.920062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.200866] Lustre: lustre-OST0000: new disk, initializing [ 6456.203751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6456.210166] Lustre: 158982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6456.371281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.607307] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6462.009733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6462.020139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6462.143053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6470.080757] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.922265] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6475.150529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6480.407715] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6480.566308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6482.926157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6482.945693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.955420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6482.971027] Lustre: Skipped 1 previous similar message [ 6482.986707] Lustre: Skipped 1 previous similar message [ 6488.038070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6488.043916] Lustre: Skipped 1 previous similar message [ 6488.622678] Lustre: server umount lustre-OST0000 complete [ 6493.153869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.184444] Lustre: Skipped 1 previous similar message [ 6498.272650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6498.284593] Lustre: Skipped 3 previous similar messages [ 6498.981649] Lustre: server umount lustre-MDT0000 complete [ 6502.877317] LustreError: 160522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159184 with bad export cookie 11342642018544763264 [ 6502.878059] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.887636] LustreError: 160522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6503.211060] Lustre: server umount lustre-MDT0001 complete [ 6507.082315] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:53:07 (1788159187) [ 6509.050696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6511.197901] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:53:11 (1788159191) [ 6519.566038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.010881] LustreError: 161394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.031067] LustreError: 161394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6520.103444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.319560] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6525.411168] LustreError: 161395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.529956] LustreError: 161394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.960685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.050650] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6547.103682] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.336113] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6560.787848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6561.190395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6561.193692] Lustre: Skipped 1 previous similar message [ 6568.647371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6576.709439] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.904270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.909511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6588.920548] Lustre: Skipped 2 previous similar messages [ 6588.937483] Lustre: Skipped 2 previous similar messages [ 6594.438898] Lustre: server umount lustre-MDT0000 complete [ 6597.710463] LustreError: 162636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.110892] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159281 with bad export cookie 11342642018544764146 [ 6599.126016] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6599.134795] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6602.842357] LustreError: 161390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6602.856746] LustreError: 161390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6605.624413] Lustre: server umount lustre-MDT0001 complete [ 6612.134378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6627.919860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.136237] LustreError: 165242:0:(llog.c:1655:llog_backup()) MGC192.168.202.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6628.141108] Lustre: 165242:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6629.858293] LustreError: 165247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.865566] LustreError: 165247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6649.317488] LustreError: 165248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.345997] LustreError: 165248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6684.761387] LustreError: 165249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.786457] LustreError: 165249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 6688.735350] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6688.742447] Lustre: 165242:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6729.746335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6733.993316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6734.829671] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6742.277938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.033764] LustreError: 148453:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000030a9a7bd x1875020482739712/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6744.147180] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (not set up) [ 6744.155278] Lustre: Skipped 7 previous similar messages [ 6748.635661] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6749.686658] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6756.503267] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6758.116496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6764.186764] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6765.800897] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6783.121587] Lustre: server umount lustre-MDT0000 complete [ 6785.511859] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.514144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6785.523042] LustreError: 165249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.523056] LustreError: 165249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 6785.524479] Lustre: Skipped 2 previous similar messages [ 6788.198170] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159470 with bad export cookie 11342642018544770600 [ 6788.208506] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6788.221575] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6788.691779] Lustre: server umount lustre-MDT0001 complete [ 6803.397720] Lustre: server umount lustre-OST0000 complete [ 6827.119505] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6836.819930] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6881.723921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.132172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6882.166626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6882.264795] Lustre: lustre-MDT0000: new disk, initializing [ 6882.369028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.375117] Lustre: Skipped 1 previous similar message [ 6882.390818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6887.618191] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6899.614067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.724651] Lustre: 170348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6899.974565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6905.673881] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6917.130984] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6923.205399] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6929.815887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.057580] Lustre: lustre-OST0000: new disk, initializing [ 6930.067320] Lustre: Skipped 1 previous similar message [ 6930.073196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6930.080136] Lustre: Skipped 2 previous similar messages [ 6930.089606] Lustre: 172319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6936.209617] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6940.270144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.284587] Lustre: Skipped 1 previous similar message [ 6940.289475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.437883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6945.661621] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6951.556055] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6951.844973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6957.156926] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6957.459774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6961.130205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.165147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6961.177549] Lustre: Skipped 1 previous similar message [ 6961.208487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6961.232173] Lustre: Skipped 1 previous similar message [ 6966.245752] Lustre: server umount lustre-OST0000 complete [ 6977.144546] Lustre: server umount lustre-MDT0000 complete [ 6981.607193] LustreError: 171902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.624654] LustreError: 171902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6982.324866] LustreError: 170339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159664 with bad export cookie 11342642018544787435 [ 6982.328575] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.336627] LustreError: 170339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6982.659802] Lustre: server umount lustre-MDT0001 complete [ 6992.148807] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:01:12 (1788159672) [ 6998.966785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.321592] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7013.273727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.751687] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7027.659329] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7033.314565] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7040.269437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.248218] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7055.893640] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7069.060879] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:02:29 (1788159749) [ 7074.276367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.291448] Lustre: Skipped 2 previous similar messages [ 7079.518818] Lustre: server umount lustre-OST0000 complete [ 7089.599647] Lustre: server umount lustre-MDT0000 complete [ 7093.090554] LustreError: 174720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159775 with bad export cookie 11342642018544788317 [ 7093.100628] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7093.105879] LustreError: 174720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7093.394508] Lustre: server umount lustre-MDT0001 complete [ 7103.601249] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7106.133970] Key type lgssc unregistered [ 7106.361566] LNet: 178467:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7106.371272] LNetError: 178467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7106.392262] LNet: Removed LNI 192.168.202.102@tcp [ 7107.363871] Key type .llcrypt unregistered [ 7107.365941] Key type ._llcrypt unregistered [ 7124.606874] Key type ._llcrypt registered [ 7124.609511] Key type .llcrypt registered [ 7124.725770] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7125.734334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.855341] alg: No test for adler32 (adler32-zlib) [ 7126.878438] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7127.098555] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7128.791545] Key type lgssc registered [ 7129.876198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7137.063382] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7137.085768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7152.609471] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7157.727429] LustreError: 179716:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 7162.847789] LustreError: 179716:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7162.853828] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7162.856838] LustreError: MGC192.168.202.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7162.863179] LustreError: 179716:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7162.867137] LustreError: 179716:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -22 [ 7162.870284] LustreError: 179716:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 7162.905937] Lustre: server umount lustre-OST0000 complete [ 7162.908216] LustreError: 179716:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7172.253312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.682631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.967770] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7183.846308] LustreError: 180214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.879346] LustreError: 180214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7187.084313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.531616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.131371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7199.230926] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7204.689528] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7212.773039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.973994] LustreError: 182038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.997046] LustreError: 182033:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7213.005900] Lustre: 182033:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7213.188156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.035324] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7228.699773] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7233.513172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7233.516497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.532935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.024168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.032641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.040749] Lustre: Skipped 1 previous similar message [ 7234.048537] Lustre: Skipped 1 previous similar message [ 7237.009549] Lustre: server umount lustre-MDT0000 complete [ 7239.157874] LustreError: 180214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.177857] LustreError: 180214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7240.763156] LustreError: 180196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159922 with bad export cookie 11670459344970284078 [ 7240.768275] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7240.778441] LustreError: 180196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7241.267433] Lustre: server umount lustre-MDT0001 complete [ 7255.419707] Lustre: server umount lustre-OST0000 complete [ 7264.569484] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7273.330954] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7306.164558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.377635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7306.393809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7306.458262] Lustre: lustre-MDT0000: new disk, initializing [ 7306.518712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.530479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7310.767171] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7321.058271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.199524] Lustre: 185559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7321.278088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7321.286877] Lustre: Skipped 1 previous similar message [ 7321.377420] Lustre: lustre-MDT0001: new disk, initializing [ 7321.480386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7321.528944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7321.539910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7325.833082] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7334.392332] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7340.033682] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7346.135350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7346.346771] Lustre: lustre-OST0000: new disk, initializing [ 7346.351384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7346.357660] Lustre: 187523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7346.429177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7351.836825] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7354.428054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7354.440760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7354.556500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7359.309842] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7363.265575] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7363.475899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7366.738968] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7366.892499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7370.211190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7370.225892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.232216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.242591] Lustre: Skipped 1 previous similar message [ 7370.263560] Lustre: Skipped 1 previous similar message [ 7375.158682] Lustre: server umount lustre-OST0000 complete [ 7380.448620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.469553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7380.480508] Lustre: Skipped 1 previous similar message [ 7384.905485] Lustre: server umount lustre-MDT0000 complete [ 7385.575879] LustreError: 185570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.610215] LustreError: 185570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7388.123084] LustreError: 185549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160070 with bad export cookie 11670459344970284799 [ 7388.124583] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.136970] LustreError: 185549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7388.603807] Lustre: server umount lustre-MDT0001 complete [ 7396.398507] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:07:57 (1788160077) [ 7411.269544] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7413.592536] Key type lgssc unregistered [ 7413.853665] LNet: 190467:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7413.863229] LNetError: 190467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7413.886337] LNet: Removed LNI 192.168.202.102@tcp [ 7414.659344] Key type .llcrypt unregistered [ 7414.661717] Key type ._llcrypt unregistered [ 7429.494706] Key type ._llcrypt registered [ 7429.496995] Key type .llcrypt registered [ 7429.586910] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7430.543424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7430.554110] alg: No test for adler32 (adler32-zlib) [ 7431.550239] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7431.759722] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7433.407320] Key type lgssc registered [ 7434.325582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7440.279659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7440.301048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.644573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7445.457838] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7451.117580] LustreError: 191684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7451.133517] LustreError: 191684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7452.470302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.854522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7456.804595] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7463.276774] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7467.980849] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7474.890273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7475.029611] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7475.031415] LustreError: 193548:0:(llog.c:1696:llog_backup()) MGC192.168.202.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7475.035615] Lustre: 193548:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7475.112018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7481.071584] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7488.240385] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7491.553361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7491.559995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7491.583780] Lustre: Skipped 1 previous similar message [ 7496.682891] LustreError: 191680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7496.693831] LustreError: 191680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7496.854341] Lustre: server umount lustre-MDT0000 complete [ 7500.518246] LustreError: 191664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160182 with bad export cookie 15857845705023541413 [ 7500.531456] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7500.543994] LustreError: 191664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7500.750931] Lustre: server umount lustre-MDT0001 complete [ 7513.237808] Lustre: server umount lustre-OST0000 complete [ 7522.956924] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7533.345204] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7571.012119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.207375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7571.235240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7571.320570] Lustre: lustre-MDT0000: new disk, initializing [ 7571.394243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.416346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7575.839837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7586.091113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.226676] Lustre: 197070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7586.285592] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7586.288060] Lustre: Skipped 1 previous similar message [ 7586.291252] LustreError: 197854:0:(llog.c:1696:llog_backup()) MGC192.168.202.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7586.297858] LustreError: 197854:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7586.302952] Lustre: 197854:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7586.323508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7586.331864] Lustre: Skipped 1 previous similar message [ 7586.413456] Lustre: lustre-MDT0001: new disk, initializing [ 7586.570064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.610142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7586.619158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7590.634814] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7598.497238] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7603.193546] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7609.301839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.614111] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7609.619333] Lustre: Skipped 1 previous similar message [ 7609.626569] LustreError: 199016:0:(llog.c:1696:llog_backup()) MGC192.168.202.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7609.633155] LustreError: 199016:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7609.637851] Lustre: 199016:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7609.642938] Lustre: 199016:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7609.652942] Lustre: lustre-OST0000: new disk, initializing [ 7609.657717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7609.661333] Lustre: 199040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7609.760916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.482803] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7616.598580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7616.609886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7616.706991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7624.040562] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7628.254831] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7628.422469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7632.445831] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7632.619817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7637.476066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.484130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7637.495604] Lustre: Skipped 1 previous similar message [ 7637.502598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7637.525403] Lustre: Skipped 3 previous similar messages [ 7640.590914] Lustre: server umount lustre-OST0000 complete [ 7647.712383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.713941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7647.723553] Lustre: Skipped 2 previous similar messages [ 7647.753610] Lustre: Skipped 1 previous similar message [ 7650.650422] Lustre: server umount lustre-MDT0000 complete [ 7652.846574] LustreError: 198789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.865395] LustreError: 198789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7654.476288] LustreError: 197063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160336 with bad export cookie 15857845705023542134 [ 7654.491435] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7654.497653] LustreError: 197063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7654.993767] Lustre: server umount lustre-MDT0001 complete [ 7663.872590] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:12:24 (1788160344) [ 7678.781159] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7687.185985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.566495] LustreError: 202166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.627274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7691.609762] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7692.769265] LustreError: 202167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7700.182902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.155737] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7708.369778] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7715.555853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.884973] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7729.849300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7730.067448] Lustre: lustre-OST0001: new disk, initializing [ 7730.071310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7730.076632] Lustre: 204470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7730.140736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7730.143314] Lustre: Skipped 2 previous similar messages [ 7735.759459] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7736.971872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7736.981376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7737.093937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7745.271106] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7750.177134] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7751.955035] Lustre: DEBUG MARKER: ADD OST3 [ 7756.208666] loop: module loaded [ 7756.295430] loop0: detected capacity change from 0 to 204800000 [ 7756.530648] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7756.544374] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7756.561321] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7761.816745] loop0: detected capacity change from 0 to 204800000 [ 7763.179873] Lustre: DEBUG MARKER: STOP OST3 [ 7767.110375] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7767.356126] Lustre: lustre-OST0003: new disk, initializing [ 7767.361366] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7767.366989] Lustre: 206449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7769.261932] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7769.365641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7769.380167] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7769.414638] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7771.119736] Lustre: Failing over lustre-OST0003 [ 7771.295603] LustreError: 206679:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7771.322879] Lustre: server umount lustre-OST0003 complete [ 7772.027742] LustreError: 203653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.052247] LustreError: 203653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7773.156140] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7774.699950] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7774.709806] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.119378] Lustre: DEBUG MARKER: START OST3 again [ 7787.077587] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.278534] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7787.602872] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7788.675486] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7788.680664] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7792.731646] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7811.971062] Lustre: Failing over lustre-OST0003 [ 7812.062197] Lustre: server umount lustre-OST0003 complete [ 7813.106707] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.108182] LustreError: 203653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.126122] Lustre: Skipped 2 previous similar messages [ 7813.136186] LustreError: 203653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7833.572430] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.586927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7833.587574] Lustre: Skipped 2 previous similar messages [ 7833.598895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7833.632267] Lustre: Skipped 3 previous similar messages [ 7835.061857] Lustre: server umount lustre-MDT0000 complete [ 7838.308449] LustreError: 202147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160520 with bad export cookie 15857845705023543016 [ 7838.311802] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7838.319744] LustreError: 202147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7838.693735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7838.704252] Lustre: Skipped 1 previous similar message [ 7844.948966] Lustre: server umount lustre-MDT0001 complete [ 7849.149483] Lustre: server umount lustre-OST0000 complete [ 7852.925919] Lustre: server umount lustre-OST0001 complete [ 7859.355633] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7866.893942] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7902.634286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.822773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7902.840534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7902.891458] Lustre: lustre-MDT0000: new disk, initializing [ 7902.943433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.949824] Lustre: Skipped 2 previous similar messages [ 7902.965080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7906.373579] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7916.438083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.572840] Lustre: 211936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7916.897442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7921.000659] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7930.006530] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7935.712503] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7943.026733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7943.322779] Lustre: lustre-OST0000: new disk, initializing [ 7943.326859] Lustre: Skipped 1 previous similar message [ 7943.341844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7943.346844] Lustre: Skipped 2 previous similar messages [ 7943.351730] Lustre: 213901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7945.001479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7945.020381] Lustre: Skipped 1 previous similar message [ 7945.035542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7945.184740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7950.604604] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7961.121826] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7966.111097] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7966.315305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7971.177111] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7971.396169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7975.904086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7975.909574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.917275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7975.929414] Lustre: Skipped 4 previous similar messages [ 7975.952920] Lustre: Skipped 3 previous similar messages [ 7979.708440] Lustre: server umount lustre-OST0000 complete [ 7991.042432] Lustre: server umount lustre-MDT0000 complete [ 7991.274679] LustreError: 211948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7991.293243] LustreError: 211948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7997.144942] LustreError: 211928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160679 with bad export cookie 15857845705023545312 [ 7997.153333] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7997.168123] LustreError: 211928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7998.239861] Lustre: server umount lustre-MDT0001 complete [ 8004.056224] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:18:03 (1788160683) [ 8021.330810] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8031.368678] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8066.902335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.113727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8067.133188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8067.192663] Lustre: lustre-MDT0000: new disk, initializing [ 8067.253504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.258554] Lustre: Skipped 2 previous similar messages [ 8067.274627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8071.763334] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8083.166048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.277321] Lustre: 218514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8083.588305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8088.299204] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8097.170740] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8101.973319] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8107.334081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.566553] Lustre: 220483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8109.763031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8113.597731] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8122.022183] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8125.845605] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8126.096233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8130.651606] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8130.878988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8135.138530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.149396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.166193] Lustre: Skipped 2 previous similar messages [ 8135.180141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.184570] Lustre: Skipped 2 previous similar messages [ 8138.946641] Lustre: server umount lustre-OST0000 complete [ 8144.865271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8144.881349] LustreError: Skipped 1 previous similar message [ 8149.720248] Lustre: server umount lustre-MDT0000 complete [ 8151.019391] LustreError: 220597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.059167] LustreError: 220597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8154.410392] LustreError: 218507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160836 with bad export cookie 15857845705023546194 [ 8154.413485] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8154.426838] LustreError: 218507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8154.987461] Lustre: server umount lustre-MDT0001 complete [ 8166.045691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.267292] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8182.360433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.654588] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8196.774711] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8202.536643] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8209.788521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8217.218811] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8225.348639] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8238.050580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.060400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.072668] Lustre: Skipped 3 previous similar messages [ 8238.077306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.088244] Lustre: Skipped 4 previous similar messages [ 8241.702961] Lustre: server umount lustre-OST0000 complete [ 8252.243198] Lustre: server umount lustre-MDT0000 complete [ 8256.493089] LustreError: 222680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160938 with bad export cookie 15857845705023547076 [ 8256.506755] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.889025] Lustre: server umount lustre-MDT0001 complete [ 8367.353790] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8369.788251] Key type lgssc unregistered [ 8370.099810] LNet: 226362:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8370.107483] LNetError: 226362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8370.144101] LNet: Removed LNI 192.168.202.102@tcp [ 8370.946336] Key type .llcrypt unregistered [ 8370.949790] Key type ._llcrypt unregistered [ 8393.971983] Key type ._llcrypt registered [ 8393.974637] Key type .llcrypt registered [ 8394.110543] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8409.942180] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8411.316703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8411.432488] alg: No test for adler32 (adler32-zlib) [ 8412.467490] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8412.656745] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8414.295180] Key type lgssc registered [ 8415.526389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8448.346561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8448.386978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.720885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8449.762243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8449.859996] Lustre: lustre-MDT0000: new disk, initializing [ 8449.980835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.004624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8454.193976] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8463.759430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.926991] Lustre: 229716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8464.003366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8464.010078] Lustre: Skipped 1 previous similar message [ 8464.151871] Lustre: lustre-MDT0001: new disk, initializing [ 8464.301488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.363501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8464.377780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8469.358693] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8479.506607] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8485.774531] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8492.478072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.781425] Lustre: lustre-OST0000: new disk, initializing [ 8492.784918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8492.789948] Lustre: 231684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8492.898067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8500.392316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8502.395235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8502.405829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8502.568782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8510.786801] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8515.072483] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8515.210396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8519.367739] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8519.552378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8523.238818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.243272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8523.264641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8523.272953] LustreError: Skipped 1 previous similar message [ 8527.343939] Lustre: server umount lustre-OST0000 complete [ 8533.485205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.491759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8533.497526] Lustre: Skipped 2 previous similar messages [ 8533.503649] Lustre: Skipped 1 previous similar message [ 8537.356163] Lustre: server umount lustre-MDT0000 complete [ 8541.429986] LustreError: 229707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161223 with bad export cookie 6187625910523389477 [ 8541.434921] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8541.443049] LustreError: 229707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8541.949817] Lustre: server umount lustre-MDT0001 complete [ 8559.205511] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8568.158650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.647834] LustreError: 234616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.674123] LustreError: 234616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8568.751225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8572.974694] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8573.921290] LustreError: 234617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8578.019086] LustreError: 234616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8580.957968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.319121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8586.257320] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8589.299859] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8596.180663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8596.443306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8602.745211] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8612.055547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8612.250519] Lustre: lustre-OST0001: new disk, initializing [ 8612.254567] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8612.261368] Lustre: 236924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8618.942431] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8620.165917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8620.186037] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8620.261899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8628.780892] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8634.263846] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8645.379120] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:28:46 (1788161326) [ 8647.654437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8647.661777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.680719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8653.896423] Lustre: server umount lustre-MDT0000 complete [ 8654.309166] LustreError: 234617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8654.328339] LustreError: 234617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8657.780466] LustreError: 235749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161339 with bad export cookie 6187625910523390359 [ 8657.784195] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8657.793333] LustreError: 235749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8658.283840] Lustre: server umount lustre-MDT0001 complete [ 8673.782060] Lustre: server umount lustre-OST0000 complete [ 8675.690943] Lustre: 227939:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161341/real 1788161341] req@ffff90b63e4c8700 x1875022807411840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788161357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8675.785453] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.810671] Lustre: Skipped 3 previous similar messages [ 8678.387423] Lustre: server umount lustre-OST0001 complete [ 8686.104719] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8698.062083] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8736.138906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.374973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8736.392924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.485576] Lustre: lustre-MDT0000: new disk, initializing [ 8736.552885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.560803] Lustre: Skipped 1 previous similar message [ 8736.579253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8742.077231] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8755.200649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.376424] Lustre: 240867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8755.440658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8755.443750] Lustre: Skipped 1 previous similar message [ 8755.520255] Lustre: lustre-MDT0001: new disk, initializing [ 8755.700304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8755.715971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8760.331919] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8769.609167] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8775.434621] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8781.167191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8781.398662] Lustre: lustre-OST0000: new disk, initializing [ 8781.403235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8781.411450] Lustre: 242835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8781.485944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.489637] Lustre: Skipped 1 previous similar message [ 8783.094823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8783.107475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8783.174570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8786.606516] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8796.621735] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8801.437861] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8801.608934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8806.251056] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8806.433863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8808.943499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8808.952436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.002290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8809.021122] Lustre: Skipped 5 previous similar messages [ 8814.052342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8814.061879] Lustre: Skipped 1 previous similar message [ 8823.263158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8823.673903] Lustre: server umount lustre-OST0000 complete [ 8829.431127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.460300] Lustre: Skipped 1 previous similar message [ 8829.471686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.481450] Lustre: Skipped 3 previous similar messages [ 8834.401640] Lustre: server umount lustre-MDT0000 complete [ 8834.533405] LustreError: 240875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8834.556420] LustreError: 240875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8839.928716] LustreError: 240861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161521 with bad export cookie 6187625910523392494 [ 8839.937284] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8839.949925] LustreError: 240861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8840.547709] Lustre: server umount lustre-MDT0001 complete [ 8850.157258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.840187] LustreError: 245057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.876932] LustreError: 245057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8850.948672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.208656] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8866.030803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.289506] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8880.338691] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8888.375722] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8897.165698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8903.978588] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8911.087934] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9059.304934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.315572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9059.344676] Lustre: Skipped 2 previous similar messages [ 9064.429962] LustreError: 246879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.444395] LustreError: 246879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9064.586551] Lustre: server umount lustre-OST0000 complete [ 9083.359675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9083.537447] Lustre: server umount lustre-MDT0000 complete [ 9086.762500] LustreError: 245039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161768 with bad export cookie 6187625910523393376 [ 9086.770435] LustreError: 245039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9086.776248] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9087.091673] Lustre: server umount lustre-MDT0001 complete [ 9178.705896] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9181.685705] Key type lgssc unregistered [ 9182.075860] LNet: 248662:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9182.083577] LNetError: 248662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9182.099633] LNet: Removed LNI 192.168.202.102@tcp [ 9182.756138] Key type .llcrypt unregistered [ 9182.758445] Key type ._llcrypt unregistered [ 9210.319374] Key type ._llcrypt registered [ 9210.321446] Key type .llcrypt registered [ 9210.420859] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9230.962540] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9232.466439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9232.490320] alg: No test for adler32 (adler32-zlib) [ 9233.472830] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9233.811248] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9235.487515] Key type lgssc registered [ 9236.840898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9285.825412] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9296.184550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9296.235387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9297.552937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9297.592629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9297.737743] Lustre: lustre-MDT0000: new disk, initializing [ 9297.877690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.928327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9302.456520] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9314.356346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9314.497289] Lustre: 252737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9314.548851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9314.553483] Lustre: Skipped 1 previous similar message [ 9314.601757] Lustre: lustre-MDT0001: new disk, initializing [ 9314.678939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9314.709482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9314.719916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9319.141372] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9324.589352] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9331.912586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9332.160638] Lustre: lustre-OST0000: new disk, initializing [ 9332.165886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9332.170654] Lustre: 254368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9332.262321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.310495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9336.324079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9336.495635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9337.982598] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9349.414089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9349.590351] Lustre: lustre-OST0001: new disk, initializing [ 9349.593214] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9349.598429] Lustre: 255241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9349.701522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9357.236417] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9357.554740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9357.564389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9357.676653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9370.071209] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9380.043501] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9393.637489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9393.644075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9393.664434] Lustre: Skipped 3 previous similar messages [ 9393.668571] Lustre: Skipped 2 previous similar messages [ 9398.764903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9398.771125] Lustre: Skipped 1 previous similar message [ 9398.930668] Lustre: server umount lustre-MDT0000 complete [ 9402.747167] LustreError: 252729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162084 with bad export cookie 8560226033057365206 [ 9402.757864] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9402.763552] LustreError: 252729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9403.285064] Lustre: server umount lustre-MDT0001 complete [ 9417.296241] Lustre: server umount lustre-OST0000 complete [ 9419.232113] Lustre: 250244:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788162085/real 1788162085] req@ffff90b625087800 x1875023668263168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788162101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9419.254556] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.835900] Lustre: server umount lustre-OST0001 complete [ 9436.532639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9447.417116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9456.162617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9464.696028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9487.404523] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9501.574701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.605740] Lustre: lustre-MDT0000: reset Object Index mappings [ 9502.518146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.689171] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9507.828289] LustreError: 260265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9507.866705] LustreError: 260265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9512.947637] LustreError: 260264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.418665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.456507] Lustre: lustre-MDT0001: reset Object Index mappings [ 9517.048986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9522.894616] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9525.967926] Lustre: 261406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9532.608239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.523407] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9541.181627] LustreError: 261761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9546.221597] LustreError: 262212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9547.309603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9547.818728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9548.003132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9548.010310] Lustre: Skipped 1 previous similar message [ 9553.458591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9554.526903] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9561.955330] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9565.780298] Lustre: 263276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9573.365492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9573.373662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9573.403704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9573.414649] Lustre: Skipped 3 previous similar messages [ 9573.642151] Lustre: server umount lustre-MDT0000 complete [ 9573.881175] LustreError: 260258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9573.916318] LustreError: 260258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9577.634788] LustreError: 260246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162259 with bad export cookie 8560226033057371996 [ 9577.637945] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.641911] LustreError: 260246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9578.002820] Lustre: server umount lustre-MDT0001 complete [ 9591.186944] Lustre: server umount lustre-OST0000 complete [ 9595.104291] Lustre: 250245:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788162261/real 1788162261] req@ffff90b62b84ca80 x1875023668311168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788162277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9595.146318] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.154853] Lustre: Skipped 3 previous similar messages [ 9595.598553] Lustre: server umount lustre-OST0001 complete [ 9617.760394] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:44:57 (1788162297) [ 9642.877315] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9654.685726] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9695.031994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.305144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9695.343144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9695.460481] Lustre: lustre-MDT0000: new disk, initializing [ 9695.573242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.599040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9701.095621] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9712.192214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.398229] Lustre: 267531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9712.493468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9712.496714] Lustre: Skipped 1 previous similar message [ 9712.608656] Lustre: lustre-MDT0001: new disk, initializing [ 9712.751124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9712.782768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9717.974782] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9727.843911] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9733.530240] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9740.736168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9741.039980] Lustre: lustre-OST0000: new disk, initializing [ 9741.044520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9741.051730] Lustre: 269500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9742.183433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9742.197192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9742.414242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9747.810750] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9757.000600] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9762.287273] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9762.549394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9766.868068] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9767.062659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9773.038062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9773.039289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9773.048494] Lustre: Skipped 1 previous similar message [ 9773.053387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9773.056629] LustreError: Skipped 1 previous similar message [ 9773.060659] Lustre: Skipped 1 previous similar message [ 9778.145201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9778.156465] Lustre: Skipped 1 previous similar message [ 9783.263221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9783.523042] Lustre: server umount lustre-OST0000 complete [ 9788.386884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.392485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9788.407547] Lustre: Skipped 1 previous similar message [ 9788.427328] Lustre: Skipped 3 previous similar messages [ 9794.804410] Lustre: server umount lustre-MDT0000 complete [ 9798.624963] LustreError: 267544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.644581] LustreError: 267544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9799.779689] LustreError: 267522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162481 with bad export cookie 8560226033057374089 [ 9799.786296] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.789285] LustreError: 267522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9800.220853] Lustre: server umount lustre-MDT0001 complete [ 9824.383763] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9836.146058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.587479] LustreError: 272454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9836.634994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.637526] Lustre: Skipped 2 previous similar messages [ 9841.469781] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9851.018130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9856.405416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9859.469927] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9868.804427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9878.961650] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9889.279124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9889.488676] Lustre: lustre-OST0001: new disk, initializing [ 9889.492665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9889.498880] Lustre: 274764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9890.768260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9890.785336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9890.950301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9897.497730] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9909.482814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9914.767169] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9921.114587] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9921.191762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9931.755309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9931.764280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9931.780057] Lustre: Skipped 2 previous similar messages [ 9931.782371] Lustre: Skipped 3 previous similar messages [ 9936.002676] Lustre: server umount lustre-MDT0000 complete [ 9936.896448] LustreError: 272455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9936.918860] LustreError: 272455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9941.705666] LustreError: 274763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162623 with bad export cookie 8560226033057374971 [ 9941.715780] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9941.732441] LustreError: 274763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9942.229677] Lustre: server umount lustre-MDT0001 complete [ 9949.974847] Lustre: server umount lustre-OST0000 complete [ 9954.019841] Lustre: server umount lustre-OST0001 complete [ 9973.770618] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9986.263254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.682333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.686569] Lustre: Skipped 3 previous similar messages [ 9991.253770] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10001.240236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.379076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10001.381983] Lustre: Skipped 4 previous similar messages [10006.931811] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10010.363906] Lustre: 278958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10019.009436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10026.751236] LustreError: 279312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10026.792950] LustreError: 279312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10028.466385] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10031.878267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10039.487536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10044.935883] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10047.194066] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10055.866451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10060.230507] Lustre: 280834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10066.373721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10066.582351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10076.664980] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:52:36 (1788162756) [10093.540270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10093.543744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10093.569077] Lustre: Skipped 3 previous similar messages [10093.582268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10098.470945] Lustre: server umount lustre-MDT0000 complete [10102.122370] LustreError: 277799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162784 with bad export cookie 8560226033057381306 [10102.127679] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10102.134984] LustreError: 277799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10108.560848] Lustre: server umount lustre-MDT0001 complete [10118.430717] Lustre: server umount lustre-OST0000 complete [10122.154808] Lustre: server umount lustre-OST0001 complete [10129.186242] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [10140.085186] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10189.241250] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10199.039518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.334199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10199.357956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10199.418223] Lustre: lustre-MDT0000: new disk, initializing [10199.514567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10204.318189] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10214.227274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.338326] Lustre: 285074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10214.582385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10218.872212] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10231.576194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10232.023833] Lustre: 286707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10233.845355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10239.665320] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10251.293660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10251.478815] Lustre: 287577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10251.586990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10251.592191] Lustre: Skipped 6 previous similar messages [10252.713338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10252.726014] Lustre: Skipped 1 previous similar message [10252.840925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10259.220911] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10269.619615] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10273.145634] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10273.149926] Lustre: Skipped 1 previous similar message [10288.732840] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:56:08 (1788162968) [10291.682622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10291.696850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10291.721097] Lustre: Skipped 5 previous similar messages [10291.728543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10291.734538] Lustre: Skipped 8 previous similar messages [10293.025700] LustreError: 288743:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff90b6272d7a00) refcount nonzero (2) after lock cleanup; forcing cleanup. [10293.122980] Lustre: server umount lustre-MDT0000 complete [10293.736347] LustreError: 285086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10293.766181] LustreError: 285086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [10297.105182] LustreError: 285870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162979 with bad export cookie 8560226033057389202 [10297.108923] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10297.135782] LustreError: 285870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10297.617524] Lustre: server umount lustre-MDT0001 complete [10311.784903] Lustre: server umount lustre-OST0000 complete [10314.719291] Lustre: 250244:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788162980/real 1788162980] req@ffff90b5021d4000 x1875023668577152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788162996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10315.663434] Lustre: server umount lustre-OST0001 complete [10324.621433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10331.091930] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10343.425207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10349.253869] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10359.258796] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10366.094281] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10374.271879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10380.920988] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10388.963709] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10416.953570] loop1: detected capacity change from 0 to 204800000 [10417.222738] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10417.230740] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10417.247281] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10423.402661] loop1: detected capacity change from 0 to 204800000 [10430.294309] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.425331] LustreError: 289808:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs15247: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10430.456948] LustreError: 289808:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs15247 [10430.462800] LustreError: 289808:0:(mgs_handler.c:517:mgs_target_reg()) fs15247-MDT0001: Error checking target index: rc = -13 [10430.480848] LustreError: fs15247-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10430.493352] LustreError: 293566:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10430.501829] LustreError: 293566:0:(tgt_mount.c:2206:server_put_super()) no obd fs15247-MDT0001 [10430.507797] LustreError: 293566:0:(tgt_mount.c:116:server_deregister_mount()) fs15247-MDT0001 not registered [10430.568135] Lustre: server umount fs15247-MDT0001 complete [10430.576823] LustreError: 293566:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10440.932411] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10441.041074] Lustre: Setting parameter fs15247-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15247-MDT0001 [10441.088504] Lustre: srv-fs15247-MDT0001: No data found on store. Initialize space. [10441.094575] Lustre: Skipped 4 previous similar messages [10441.175596] Lustre: fs15247-MDT0001: new disk, initializing [10441.191340] Lustre: Skipped 3 previous similar messages [10441.316452] Lustre: srv-fs15247-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10441.326818] Lustre: 294034:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15247-MDT0001-mdtlov: can't create new llog: rc = -115 [10441.347319] LustreError: 294034:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15247-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10447.141348] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10455.466811] Lustre: server umount fs15247-MDT0001 complete [10466.792720] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10471.087513] loop1: detected capacity change from 0 to 204800000 [10478.341874] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.760704] Lustre: 295603:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15247-MDT0001-mdtlov: can't create new llog: rc = -115 [10478.766558] LustreError: 295603:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15247-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10483.531336] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10488.215312] Lustre: server umount fs15247-MDT0001 complete [10502.627749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10502.629947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.639896] LustreError: Skipped 1 previous similar message [10502.658697] Lustre: Skipped 4 previous similar messages [10504.073798] Lustre: server umount lustre-MDT0000 complete [10514.403730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.540395] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10519.937608] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10520.074228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10528.282049] loop1: detected capacity change from 0 to 204800000 [10535.533552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.162011] Lustre: 298049:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15247-MDT0001-mdtlov: can't create new llog: rc = -115 [10536.173535] LustreError: 298049:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15247-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10541.284029] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10547.255783] Lustre: server umount fs15247-MDT0001 complete [10550.755985] LustreError: 291655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10550.785917] LustreError: 291655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 105 previous similar messages [10573.281089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10573.291986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10573.297479] Lustre: Skipped 4 previous similar messages [10587.103144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10587.835622] Lustre: server umount lustre-OST0000 complete [10598.873278] Lustre: server umount lustre-MDT0000 complete [10603.955829] LustreError: 294579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163285 with bad export cookie 8560226033057391883 [10603.965646] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10603.974593] LustreError: 294579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10604.004890] LustreError: 250245:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [10604.861871] Lustre: server umount lustre-MDT0001 complete [10620.023877] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10623.103958] Key type lgssc unregistered [10623.459568] LNet: 300405:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10623.468784] LNetError: 300405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10623.491382] LNet: Removed LNI 192.168.202.102@tcp [10624.619153] Key type .llcrypt unregistered [10624.623610] Key type ._llcrypt unregistered [10645.616907] Key type ._llcrypt registered [10645.623507] Key type .llcrypt registered [10645.765618] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10646.973908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10647.141182] alg: No test for adler32 (adler32-zlib) [10648.412425] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10648.661371] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10650.415559] Key type lgssc registered [10651.595141] Lustre: Echo OBD driver; http://www.lustre.org/ [10659.811716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10659.842150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.393764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10666.118920] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10671.588751] LustreError: 301673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.606613] LustreError: 301673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10674.895833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.340908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10680.290719] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10688.704469] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10695.152969] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10702.588911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10703.038107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10706.100905] LustreError: 303504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.608545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10711.523678] LustreError: 303505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.675476] LustreError: 303504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.968724] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10720.749319] LustreError: 303666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10720.802129] LustreError: 303666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10730.986126] LustreError: 303502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.006363] LustreError: 303502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10739.993217] loop1: detected capacity change from 0 to 204800000 [10740.265856] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10740.281977] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10740.296531] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10746.652994] loop1: detected capacity change from 0 to 204800000 [10751.477713] LustreError: 303504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.491926] LustreError: 303504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10753.189565] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10753.389450] Lustre: fs29520-OST0000: new disk, initializing [10753.392726] Lustre: srv-fs29520-OST0000: No data found on store. Initialize space. [10753.400422] Lustre: 305412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10753.498317] Lustre: fs29520-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10762.051561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10772.177451] Lustre: server umount fs29520-OST0000 complete [10787.307858] LustreError: 303504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.341660] LustreError: 303504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [10787.358688] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10789.090081] loop1: detected capacity change from 0 to 204800000 [10796.208529] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10796.417731] Lustre: fs29520-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10802.670200] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10802.729626] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10813.902151] loop2: detected capacity change from 0 to 204800000 [10814.180800] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10814.218260] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10814.238986] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10819.853407] loop2: detected capacity change from 0 to 204800000 [10826.102566] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.202887] LustreError: fs29520-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10826.210602] LustreError: fs29520-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10826.215339] LustreError: 308456:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10826.220525] LustreError: 308456:0:(tgt_mount.c:2206:server_put_super()) no obd fs29520-OST0001 [10826.225484] LustreError: 308456:0:(tgt_mount.c:116:server_deregister_mount()) fs29520-OST0001 not registered [10826.256405] Lustre: server umount fs29520-OST0001 complete [10826.260383] LustreError: 308456:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10838.608511] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10838.909957] Lustre: fs29520-OST0001: new disk, initializing [10838.917105] Lustre: srv-fs29520-OST0001: No data found on store. Initialize space. [10838.932656] Lustre: 309007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10839.064253] Lustre: fs29520-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10845.899372] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10853.859042] LustreError: 303505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.886610] LustreError: 303505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10855.075582] Lustre: server umount fs29520-OST0000 complete [10865.866631] Lustre: server umount fs29520-OST0001 complete [10880.840427] loop1: detected capacity change from 0 to 204800000 [10888.144265] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10888.879453] Lustre: fs29520-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10897.358649] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10905.975716] loop2: detected capacity change from 0 to 204800000 [10913.526460] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10922.228916] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10932.657434] Lustre: server umount fs29520-OST0000 complete [10940.905102] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10942.570709] Lustre: server umount fs29520-OST0001 complete [10956.259284] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10956.262957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10956.272120] LustreError: Skipped 1 previous similar message [10956.298860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10956.305225] Lustre: Skipped 1 previous similar message [10961.381874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10962.387929] Lustre: server umount lustre-OST0000 complete [10967.011946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10967.022734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10967.034346] Lustre: Skipped 1 previous similar message [10967.040735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10967.047523] Lustre: Skipped 1 previous similar message [10971.617328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.623655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.633385] Lustre: Skipped 1 previous similar message [10976.737505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10976.744769] Lustre: Skipped 1 previous similar message [10981.343146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10981.532352] Lustre: server umount lustre-MDT0000 complete [10981.858034] LustreError: 301670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.879455] LustreError: 301670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 63 previous similar messages [10985.645215] LustreError: 301654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163667 with bad export cookie 12452885562012661192 [10985.678384] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10985.680283] LustreError: 301654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10985.979251] Lustre: server umount lustre-MDT0001 complete [10997.871746] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11002.322597] Key type lgssc unregistered [11002.819745] LNet: 314935:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11002.835638] LNetError: 314935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11003.881566] LNet: Removed LNI 192.168.202.102@tcp [11005.348301] Key type .llcrypt unregistered [11005.350163] Key type ._llcrypt unregistered [11044.738633] Key type ._llcrypt registered [11044.741224] Key type .llcrypt registered [11044.835063] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11062.927788] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11064.784349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11064.928135] alg: No test for adler32 (adler32-zlib) [11066.278753] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11066.656958] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11068.367192] Key type lgssc registered [11070.415169] Lustre: Echo OBD driver; http://www.lustre.org/ [11109.008473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11109.038055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11110.433890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11110.528873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11110.601255] Lustre: lustre-MDT0000: new disk, initializing [11110.679163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11110.697971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11115.556759] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11125.826425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.995196] Lustre: 318799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11126.079488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11126.086288] Lustre: Skipped 1 previous similar message [11126.182286] Lustre: lustre-MDT0001: new disk, initializing [11126.284656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11126.317257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11126.332222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11131.302710] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11140.437347] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11145.922795] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11152.721824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11153.152320] Lustre: lustre-OST0000: new disk, initializing [11153.161224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11153.174043] Lustre: 320767:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11153.324875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11159.671711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11159.682293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11159.851628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11160.283058] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11171.725495] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11173.883979] Lustre: Mounted lustre-client [11174.969595] Lustre: Unmounted lustre-client [11185.632453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11185.633959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11185.636047] LustreError: Skipped 1 previous similar message [11185.648929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11189.229706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11189.241492] Lustre: Skipped 2 previous similar messages [11191.294963] Lustre: server umount lustre-OST0000 complete [11197.925298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11197.934232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.945187] Lustre: Skipped 1 previous similar message [11197.970513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11199.460282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11201.705854] Lustre: server umount lustre-MDT0000 complete [11206.188809] LustreError: 318790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163888 with bad export cookie 15122743032398089049 [11206.190901] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11206.202020] LustreError: 318790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11206.703229] Lustre: server umount lustre-MDT0001 complete [11220.856694] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11224.298630] Key type lgssc unregistered [11235.679258] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [11235.940323] LNet: 322929:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11235.954565] LNetError: 322929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11306.663984] LNet: Removed LNI 192.168.202.102@tcp [11307.612619] Key type .llcrypt unregistered [11307.615538] Key type ._llcrypt unregistered [11327.765855] Key type ._llcrypt registered [11327.768662] Key type .llcrypt registered [11327.931537] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11347.055576] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11348.030961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11348.166965] alg: No test for adler32 (adler32-zlib) [11349.310721] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11349.584918] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11351.295247] Key type lgssc registered [11352.625684] Lustre: Echo OBD driver; http://www.lustre.org/ [11387.896579] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:14:28 (1788164068) [11396.559861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11396.597095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.982616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11398.030858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11398.166034] Lustre: lustre-MDT0000: new disk, initializing [11398.259141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11398.284512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11403.228741] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11415.353718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.505449] Lustre: 326165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11415.589810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11415.593280] Lustre: Skipped 1 previous similar message [11415.686144] Lustre: lustre-MDT0001: new disk, initializing [11415.772978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11415.806418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11415.819447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11421.363034] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11431.720837] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11438.006735] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11444.585351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11444.915685] Lustre: lustre-OST0000: new disk, initializing [11444.918891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11444.923147] Lustre: 328134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11445.002395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11452.259360] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11454.491743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11454.505454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11454.662301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11461.976930] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11480.549336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11480.549369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11480.553997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11480.556561] LustreError: Skipped 1 previous similar message [11480.568295] Lustre: Skipped 1 previous similar message [11482.780025] Lustre: server umount lustre-OST0000 complete [11487.712702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11487.718634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11487.745621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11487.758668] Lustre: Skipped 1 previous similar message [11490.787167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11490.789145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11490.836955] Lustre: Skipped 1 previous similar message [11493.984767] Lustre: server umount lustre-MDT0000 complete [11498.525448] LustreError: 326158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164180 with bad export cookie 5223597519975494730 [11498.528935] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11498.530581] LustreError: 326158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11499.196744] Lustre: server umount lustre-MDT0001 complete [11513.180054] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11516.633943] Key type lgssc unregistered [11516.939753] LNet: 330042:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11516.946984] LNetError: 330042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11516.964484] LNet: Removed LNI 192.168.202.102@tcp [11517.989643] Key type .llcrypt unregistered [11517.993477] Key type ._llcrypt unregistered [11541.349026] Key type ._llcrypt registered [11541.350537] Key type .llcrypt registered [11541.469754] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11542.791344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11542.915552] alg: No test for adler32 (adler32-zlib) [11544.026525] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11544.430442] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11546.176076] Key type lgssc registered [11547.335908] Lustre: Echo OBD driver; http://www.lustre.org/ [11555.379163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11555.395461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11556.919986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11562.189293] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11566.072120] LustreError: 331317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.094798] LustreError: 331317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11571.172937] LustreError: 331316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11571.609065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.955894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11576.493757] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11585.199828] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11591.456263] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11599.010295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11599.263166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11605.596579] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11614.641524] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11625.533677] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11627.709380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11632.781452] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [11633.638962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11633.644737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11633.655804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.218317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.229078] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11637.242342] Lustre: Skipped 1 previous similar message [11638.540670] Lustre: server umount lustre-MDT0000 complete [11642.344202] LustreError: 332670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.898942] LustreError: 331298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164324 with bad export cookie 12172941350732632017 [11642.908038] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11642.997642] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [11643.010716] Lustre: Skipped 3 previous similar messages [11647.463700] LustreError: 331313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11647.475860] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.497682] LustreError: 331313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11647.530710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11649.517534] Lustre: server umount lustre-MDT0001 complete [11653.750576] Lustre: server umount lustre-OST0000 complete [11664.128456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.409984] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11670.738872] Lustre: 335497:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.102@tcp [11672.633685] Lustre: server umount MGS complete [11681.250469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.862115] LustreError: 336070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.881603] LustreError: 336070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11681.941681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11686.888863] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11691.504609] LustreError: 336071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11691.530748] LustreError: 336071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11695.229638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.539569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11699.694422] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11708.034443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.413069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11714.516170] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11724.740333] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11727.115823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11736.132418] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11738.237412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11746.272362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11746.281656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11746.302748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11746.307100] Lustre: Skipped 1 previous similar message [11747.309634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11750.456525] Lustre: server umount lustre-OST0000 complete [11757.024609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11757.039831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.058301] Lustre: Skipped 1 previous similar message [11761.876697] Lustre: server umount lustre-MDT0000 complete [11762.656625] LustreError: 336070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.679446] LustreError: 336070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11765.879900] LustreError: 336050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164447 with bad export cookie 12172941350732632850 [11765.888363] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11765.896581] LustreError: 336050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11766.610580] Lustre: server umount lustre-MDT0001 complete [11778.533789] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11781.727582] Key type lgssc unregistered [11782.106688] LNet: 339508:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11782.118331] LNetError: 339508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11783.145836] LNet: Removed LNI 192.168.202.102@tcp [11784.198106] Key type .llcrypt unregistered [11784.199680] Key type ._llcrypt unregistered [11803.138935] Key type ._llcrypt registered [11803.143347] Key type .llcrypt registered [11803.288466] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11820.736657] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11821.853812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11822.011838] alg: No test for adler32 (adler32-zlib) [11823.169517] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11823.423719] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11825.089281] Key type lgssc registered [11826.159465] Lustre: Echo OBD driver; http://www.lustre.org/ [11868.454616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11868.530375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.835529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11869.868731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11870.020372] Lustre: lustre-MDT0000: new disk, initializing [11870.207120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11870.245642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11877.762614] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11891.347781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11891.529802] Lustre: 342671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11891.597549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11891.603282] Lustre: Skipped 1 previous similar message [11891.690488] Lustre: lustre-MDT0001: new disk, initializing [11891.807189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11891.868910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11891.897776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11898.138254] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11910.076169] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11917.995891] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11926.613765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11926.963154] Lustre: lustre-OST0000: new disk, initializing [11926.973352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11926.986372] Lustre: 344643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11927.127493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11934.043887] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11934.254593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11934.261819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11934.383857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11946.230721] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11951.812537] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11952.056240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11957.450445] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11957.714944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11965.411134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.428817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11965.430473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11965.448420] Lustre: Skipped 1 previous similar message [11965.488430] LustreError: Skipped 1 previous similar message [11967.150356] Lustre: server umount lustre-OST0000 complete [11973.603918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11973.617786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11973.635589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11973.638354] Lustre: Skipped 1 previous similar message [11975.649218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.650252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11975.664535] Lustre: Skipped 1 previous similar message [11979.755565] Lustre: server umount lustre-MDT0000 complete [11983.764498] LustreError: 342664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164665 with bad export cookie 16196128402744000369 [11983.781232] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11983.794611] LustreError: 342664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11984.150526] Lustre: server umount lustre-MDT0001 complete [11994.565497] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:24:34 (1788164674) [12012.386946] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12026.764491] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12073.508501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.797184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12073.853169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12073.930374] Lustre: lustre-MDT0000: new disk, initializing [12073.997912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12074.021897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12079.521243] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12091.134194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12091.292581] Lustre: 349405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12091.333400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12091.336402] Lustre: Skipped 1 previous similar message [12091.538674] Lustre: lustre-MDT0001: new disk, initializing [12091.676345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.751175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12091.765935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12096.655146] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12106.645417] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12113.265951] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12120.491049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12120.753713] Lustre: lustre-OST0000: new disk, initializing [12120.757300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12120.763209] Lustre: 351376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12120.862043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12126.476169] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12130.384475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12130.389749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12130.586129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12137.445306] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12176.866624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12176.867286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.882186] LustreError: Skipped 1 previous similar message [12176.885565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12176.909495] Lustre: Skipped 1 previous similar message [12181.466515] Lustre: server umount lustre-OST0000 complete [12187.105324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12187.129746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12187.141879] Lustre: Skipped 1 previous similar message [12187.165776] Lustre: Skipped 2 previous similar messages [12191.654791] Lustre: server umount lustre-MDT0000 complete [12192.224498] LustreError: 349413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.256693] LustreError: 349413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12195.886598] LustreError: 349398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164877 with bad export cookie 16196128402744001251 [12195.898943] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12195.917058] LustreError: 349398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12196.279955] Lustre: server umount lustre-MDT0001 complete [12206.180132] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12208.610615] Key type lgssc unregistered [12208.876243] LNet: 353553:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12208.880643] LNetError: 353553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12208.892629] LNet: Removed LNI 192.168.202.102@tcp [12209.737415] Key type .llcrypt unregistered [12209.744585] Key type ._llcrypt unregistered [12303.142511] Key type ._llcrypt registered [12303.149611] Key type .llcrypt registered [12303.306698] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12303.738342] Key type .llcrypt unregistered [12303.745770] Key type ._llcrypt unregistered [12377.225872] Key type ._llcrypt registered [12377.228260] Key type .llcrypt registered [12377.330994] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12377.648265] Key type .llcrypt unregistered [12377.649952] Key type ._llcrypt unregistered [12408.826189] Key type ._llcrypt registered [12408.828133] Key type .llcrypt registered [12408.945269] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12409.244286] Key type .llcrypt unregistered [12409.254785] Key type ._llcrypt unregistered [12437.919355] Key type ._llcrypt registered [12437.923148] Key type .llcrypt registered [12438.009585] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12438.290830] Key type .llcrypt unregistered [12438.293483] Key type ._llcrypt unregistered [12532.900926] Key type ._llcrypt registered [12532.903868] Key type .llcrypt registered [12533.125885] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12533.549317] Key type .llcrypt unregistered [12533.552472] Key type ._llcrypt unregistered [12574.544661] Key type ._llcrypt registered [12574.562238] Key type .llcrypt registered [12574.694659] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12575.075633] Key type .llcrypt unregistered [12575.077186] Key type ._llcrypt unregistered [12609.923402] Key type ._llcrypt registered [12609.926609] Key type .llcrypt registered [12610.086413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12610.477956] Key type .llcrypt unregistered [12610.479667] Key type ._llcrypt unregistered [12655.347388] Key type ._llcrypt registered [12655.357484] Key type .llcrypt registered [12655.514735] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12655.903639] Key type .llcrypt unregistered [12655.905300] Key type ._llcrypt unregistered [12688.116802] Key type ._llcrypt registered [12688.121813] Key type .llcrypt registered [12688.264927] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12688.587293] Key type .llcrypt unregistered [12688.589061] Key type ._llcrypt unregistered [12720.888953] Key type ._llcrypt registered [12720.896600] Key type .llcrypt registered [12721.074531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12721.408341] Key type .llcrypt unregistered [12721.409991] Key type ._llcrypt unregistered [12786.705507] Key type ._llcrypt registered [12786.715443] Key type .llcrypt registered [12786.863880] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12787.195295] Key type .llcrypt unregistered [12787.199628] Key type ._llcrypt unregistered [12819.723837] Key type ._llcrypt registered [12819.725850] Key type .llcrypt registered [12819.811679] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12820.189421] Key type .llcrypt unregistered [12820.203816] Key type ._llcrypt unregistered [12852.043662] Key type ._llcrypt registered [12852.045838] Key type .llcrypt registered [12852.120979] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12852.468710] Key type .llcrypt unregistered [12852.471553] Key type ._llcrypt unregistered [12906.664449] Key type ._llcrypt registered [12906.668607] Key type .llcrypt registered [12906.786622] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12921.564316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12922.660793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12922.933981] alg: No test for adler32 (adler32-zlib) [12924.092326] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12924.351836] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12926.119302] Key type lgssc registered [12927.097589] Lustre: Echo OBD driver; http://www.lustre.org/ [12964.460177] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12964.485839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.807509] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12965.847996] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12965.978521] Lustre: tst250fs-MDT0000: new disk, initializing [12966.057431] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12966.075173] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12970.160994] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12979.069823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12979.296618] Lustre: 373866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12979.424943] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12979.440114] Lustre: Skipped 1 previous similar message [12979.559527] Lustre: tst250fs-MDT0001: new disk, initializing [12979.680047] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12979.744294] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12979.767818] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12984.698743] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12990.226788] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12997.910270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12998.167472] Lustre: tst250fs-OST0000: new disk, initializing [12998.172329] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12998.185955] Lustre: 375499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12998.301885] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13004.449255] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13004.469354] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13004.669888] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13007.213236] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13019.762882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13019.940044] Lustre: tst250fs-OST0001: new disk, initializing [13019.943804] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13019.948683] Lustre: 376371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13020.043476] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13025.874804] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13025.882069] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13025.984567] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13026.804888] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13032.559695] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13035.310254] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13035.312049] Lustre: Skipped 2 previous similar messages [13056.482369] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13056.501340] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13056.541214] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13057.007155] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13057.033380] Lustre: Skipped 2 previous similar messages [13057.049127] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13057.057113] Lustre: Skipped 2 previous similar messages [13062.121503] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13062.135651] Lustre: Skipped 2 previous similar messages [13062.371964] Lustre: server umount tst250fs-MDT0000 complete [13066.508586] LustreError: 373859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165748 with bad export cookie 1097259839432882243 [13066.522977] LustreError: 373859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13066.523277] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13066.998333] Lustre: server umount tst250fs-MDT0001 complete [13081.637514] Lustre: server umount tst250fs-OST0000 complete [13083.620387] Lustre: 372085:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788165749/real 1788165749] req@ffff90b60664b100 x1875027538547200/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788165765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13083.688814] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.635489] Lustre: server umount tst250fs-OST0001 complete [13104.884913] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [13113.434697] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13147.095835] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:43:47 (1788165827) [13149.641533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13168.572091] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [13179.044795] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13214.892379] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12949 sec ========= 04:44:55 (1788165895) [13216.926688] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:44:57 (1788165897) === [13220.735987] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:45:01 (1788165901) === [13266.081893] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13268.397910] Key type lgssc unregistered [13268.673630] LNet: 385503:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13268.678807] LNetError: 385503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13268.695539] LNet: Removed LNI 192.168.202.102@tcp [13269.622543] Key type .llcrypt unregistered [13269.626195] Key type ._llcrypt unregistered