[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492487292 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003299] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007807] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011076] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013100] *** VALIDATE selinux *** [ 0.022132] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028068] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030110] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033110] *** VALIDATE tmpfs *** [ 0.034544] *** VALIDATE proc *** [ 0.036252] *** VALIDATE cgroup *** [ 0.037012] *** VALIDATE cgroup2 *** [ 0.038274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045500] debug: unmapping init [mem 0xffffffff92c59000-0xffffffff92c60fff] [ 0.047197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048777] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054020] ... fixed-purpose events: 3 [ 0.055018] ... event mask: 000000070000000f [ 0.057315] rcu: Hierarchical SRCU implementation. [ 0.059489] smp: Bringing up secondary CPUs ... [ 0.060419] x86: Booting SMP configuration: [ 0.061031] .... node #0, CPUs: #1 #2 #3 [ 0.065135] smp: Brought up 1 node, 4 CPUs [ 0.067020] smpboot: Max logical packages: 1 [ 0.068022] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.226033] node 0 deferred pages initialised in 155ms [ 0.230013] devtmpfs: initialized [ 0.231274] x86/mm: Memory block size: 128MB [ 0.234071] gcov: version magic: 0x41383552 [ 0.237399] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239000] pinctrl core: initialized pinctrl subsystem [ 0.242246] [ 0.242772] ************************************************************* [ 0.245013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247015] ** ** [ 0.250028] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.253013] ** ** [ 0.254017] ** This means that this kernel is built to expose internal ** [ 0.256015] ** IOMMU data structures, which may compromise security on ** [ 0.258019] ** your system. ** [ 0.261021] ** ** [ 0.263015] ** If you see this message and you are not debugging the ** [ 0.265018] ** kernel, report this immediately to your vendor! ** [ 0.267016] ** ** [ 0.269028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272019] ************************************************************* [ 0.274817] NET: Registered protocol family 16 [ 0.277753] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.280086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.283087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.287142] cpuidle: using governor menu [ 0.288689] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.290529] PCI: Using configuration type 1 for base access [ 0.292141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.301144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.304041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308166] cryptd: max_cpu_qlen set to 1000 [ 0.313197] ACPI: Added _OSI(Module Device) [ 0.315009] ACPI: Added _OSI(Processor Device) [ 0.316018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322317] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.328572] ACPI: Interpreter enabled [ 0.330066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.331013] ACPI: Using IOAPIC for interrupt routing [ 0.332250] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.335331] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.344863] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.347062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.352081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359815] acpiphp: Slot [2] registered [ 0.361293] acpiphp: Slot [5] registered [ 0.363178] acpiphp: Slot [6] registered [ 0.365199] acpiphp: Slot [7] registered [ 0.367169] acpiphp: Slot [8] registered [ 0.368209] acpiphp: Slot [9] registered [ 0.369124] acpiphp: Slot [10] registered [ 0.371148] acpiphp: Slot [3] registered [ 0.372163] acpiphp: Slot [4] registered [ 0.375159] acpiphp: Slot [11] registered [ 0.377122] acpiphp: Slot [12] registered [ 0.378161] acpiphp: Slot [13] registered [ 0.380114] acpiphp: Slot [14] registered [ 0.381137] acpiphp: Slot [15] registered [ 0.383164] acpiphp: Slot [16] registered [ 0.385144] acpiphp: Slot [17] registered [ 0.387155] acpiphp: Slot [18] registered [ 0.388142] acpiphp: Slot [19] registered [ 0.390114] acpiphp: Slot [20] registered [ 0.391102] acpiphp: Slot [21] registered [ 0.392141] acpiphp: Slot [22] registered [ 0.394133] acpiphp: Slot [23] registered [ 0.396147] acpiphp: Slot [24] registered [ 0.397133] acpiphp: Slot [25] registered [ 0.399204] acpiphp: Slot [26] registered [ 0.401169] acpiphp: Slot [27] registered [ 0.403140] acpiphp: Slot [28] registered [ 0.404116] acpiphp: Slot [29] registered [ 0.406153] acpiphp: Slot [30] registered [ 0.407119] acpiphp: Slot [31] registered [ 0.409068] PCI host bridge to bus 0000:00 [ 0.410017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.415028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.419031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423056] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426219] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435833] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.441601] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.444026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.447022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.453732] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.455971] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.459065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464051] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.470019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.484030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.490032] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.497072] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.504029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.512021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.538995] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.555021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.577021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.588225] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.599020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.607020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.629024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.641114] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.647028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.653025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.670022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.686000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.695027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.706023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.723020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.734228] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.741023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.748024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.773021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783170] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786459] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791422] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.794244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799154] iommu: Default domain type: Passthrough [ 0.804000] SCSI subsystem initialized [ 0.806340] ACPI: bus type USB registered [ 0.808180] usbcore: registered new interface driver usbfs [ 0.810136] usbcore: registered new interface driver hub [ 0.813146] usbcore: registered new device driver usb [ 0.815255] pps_core: LinuxPPS API ver. 1 registered [ 0.817018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.820113] PTP clock support registered [ 0.823259] EDAC MC: Ver: 3.0.0 [ 0.826359] PCI: Using ACPI for IRQ routing [ 0.827546] NetLabel: Initializing [ 0.829012] NetLabel: domain hash size = 128 [ 0.830016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.832084] NetLabel: unlabeled traffic allowed by default [ 0.834131] vgaarb: loaded [ 0.836268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.838014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.848962] clocksource: Switched to clocksource kvm-clock [ 0.959646] VFS: Disk quotas dquot_6.6.0 [ 0.961282] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963686] *** VALIDATE ramfs *** [ 0.964909] *** VALIDATE hugetlbfs *** [ 0.966691] pnp: PnP ACPI init [ 0.969130] pnp: PnP ACPI: found 6 devices [ 0.988912] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.994043] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.997508] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.000799] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.003531] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.005614] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009173] NET: Registered protocol family 2 [ 1.012099] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.016907] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.020427] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.025871] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.029812] TCP: Hash tables configured (established 65536 bind 65536) [ 1.033560] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.037105] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040580] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.044809] NET: Registered protocol family 1 [ 1.049049] RPC: Registered named UNIX socket transport module. [ 1.051746] RPC: Registered udp transport module. [ 1.053825] RPC: Registered tcp transport module. [ 1.055751] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.058485] NET: Registered protocol family 44 [ 1.060513] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.062849] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.065214] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.067696] PCI: CLS 0 bytes, default 64 [ 1.069320] Unpacking initramfs... [ 2.493628] debug: unmapping init [mem 0xffffa060fcc54000-0xffffa060fffbffff] [ 2.497842] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.500389] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.503507] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.005296] Initialise system trusted keyrings [ 3.007407] Key type blacklist registered [ 3.009413] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.021850] zbud: loaded [ 3.025382] *** VALIDATE nfs *** [ 3.026697] *** VALIDATE nfs4 *** [ 3.028787] pstore: using deflate compression [ 3.032801] Platform Keyring initialized [ 3.140309] NET: Registered protocol family 38 [ 3.143257] Key type asymmetric registered [ 3.149338] Asymmetric key parser 'x509' registered [ 3.152081] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.157615] io scheduler mq-deadline registered [ 3.160194] io scheduler kyber registered [ 3.162273] io scheduler bfq registered [ 3.164495] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.167848] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.170969] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.174092] ACPI: Power Button [PWRF] [ 3.179977] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.186859] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.199632] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.206702] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.221863] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.251928] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.285497] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.291660] Non-volatile memory driver v1.3 [ 3.293089] Linux agpgart interface v0.103 [ 3.326742] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.330079] vda: detected capacity change from 0 to 74694656 [ 3.345427] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.348820] vdb: detected capacity change from 0 to 1073741824 [ 3.365670] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.369106] vdc: detected capacity change from 0 to 2621440000 [ 3.383690] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.386271] vdd: detected capacity change from 0 to 2621440000 [ 3.399285] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.401723] vde: detected capacity change from 0 to 4294967296 [ 3.414536] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.416772] vdf: detected capacity change from 0 to 4294967296 [ 3.424479] libphy: Fixed MDIO Bus: probed [ 3.429986] usbcore: registered new interface driver usbserial_generic [ 3.432899] usbserial: USB Serial support registered for generic [ 3.435660] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.440674] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.442747] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.445248] mousedev: PS/2 mouse device common for all mice [ 3.447978] rtc_cmos 00:05: RTC can wake from S4 [ 3.450952] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.451326] rtc_cmos 00:05: registered as rtc0 [ 3.458907] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.459054] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.465366] intel_pstate: CPU model not supported [ 3.468109] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.472405] hid: raw HID events driver (C) Jiri Kosina [ 3.475311] usbcore: registered new interface driver usbhid [ 3.477644] usbhid: USB HID core driver [ 3.479570] drop_monitor: Initializing network drop monitor service [ 3.481917] Initializing XFRM netlink socket [ 3.484215] NET: Registered protocol family 10 [ 3.486936] Segment Routing with IPv6 [ 3.488488] NET: Registered protocol family 17 [ 3.490756] mpls_gso: MPLS GSO support [ 3.495640] RAS: Correctable Errors collector initialized. [ 3.497969] AVX version of gcm_enc/dec engaged. [ 3.499823] AES CTR mode by8 optimization enabled [ 3.576888] sched_clock: Marking stable (3576865313, 0)->(4537852917, -960987604) [ 3.582177] registered taskstats version 1 [ 3.584334] Loading compiled-in X.509 certificates [ 3.586216] zswap: loaded using pool lzo/zbud [ 3.608426] Key type big_key registered [ 3.620242] Key type encrypted registered [ 3.622079] ima: No TPM chip found, activating TPM-bypass! [ 3.624541] ima: Allocated hash algorithm: sha1 [ 3.626523] ima: No architecture policies found [ 3.628442] evm: Initialising EVM extended attributes: [ 3.630827] evm: security.selinux [ 3.632435] evm: security.ima [ 3.633737] evm: security.capability [ 3.635318] evm: HMAC attrs: 0x1 [ 3.638232] rtc_cmos 00:05: setting system clock to 2026-08-15 13:27:31 UTC (1786800451) [ 3.645945] debug: unmapping init [mem 0xffffffff93c03000-0xffffffff93dfffff] [ 3.649453] debug: unmapping init [mem 0xffffffff92982000-0xffffffff92c58fff] [ 3.658118] Write protecting the kernel read-only data: 28672k [ 3.661703] debug: unmapping init [mem 0xffffffff91003000-0xffffffff911fffff] [ 3.664932] debug: unmapping init [mem 0xffffffff91914000-0xffffffff919fffff] [ 3.708627] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.718449] systemd[1]: Detected virtualization kvm. [ 3.720848] systemd[1]: Detected architecture x86-64. [ 3.723234] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.751666] systemd[1]: No hostname configured. [ 3.753646] systemd[1]: Set hostname to . [ 3.755909] random: systemd: uninitialized urandom read (16 bytes read) [ 3.758487] systemd[1]: Initializing machine ID from random generator. [ 3.800532] random: ln: uninitialized urandom read (6 bytes read) [ 3.892286] random: systemd: uninitialized urandom read (16 bytes read) [ 3.894421] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.902733] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.906261] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.613317] device-mapper: uevent: version 1.0.3 [ 4.615413] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.412581] virtio_net virtio0 ens2: renamed from eth0 [ 5.423268] random: fast init done [ 5.484929] scsi host0: ata_piix [ 5.502507] scsi host1: ata_piix [ 5.504810] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.507943] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.190258] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.140539] random: crng init done [ 10.142514] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.592968] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.718705] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.011352] SELinux: Disabled at runtime. [ 12.075686] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.084791] systemd[1]: Detected virtualization kvm. [ 12.086739] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.566378] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.569898] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.575583] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.581288] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.584239] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.590485] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.594607] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 12.702853] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.025094] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.370731] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.421961] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.494389] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.520508] EDAC sbridge: Ver: 1.1.2 [ 15.161286] Key type dns_resolver registered [ 15.459339] NFS: Registering the id_resolver key type [ 15.461678] Key type id_resolver registered [ 15.463272] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-server login: [ 41.926569] libcfs: loading out-of-tree module taints kernel. [ 41.948079] Key type ._llcrypt registered [ 41.949790] Key type .llcrypt registered [ 42.005137] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 51.298590] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 52.578121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.592786] alg: No test for adler32 (adler32-zlib) [ 53.942745] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 54.793312] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 56.544185] Key type lgssc registered [ 57.957198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.709606] hrtimer: interrupt took 13757475 ns [ 73.739260] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.711543] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 126.100807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 126.138953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.447471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.477283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 127.612325] Lustre: lustre-MDT0000: new disk, initializing [ 127.700612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.727903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 131.463990] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.510477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.605747] Lustre: 6494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 144.656115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 144.660658] Lustre: Skipped 1 previous similar message [ 144.758907] Lustre: lustre-MDT0001: new disk, initializing [ 144.834540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.856733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.865233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.610088] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.611821] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 166.482324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 166.826141] Lustre: lustre-OST0000: new disk, initializing [ 166.834602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 166.849370] Lustre: 8429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 166.951499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.584348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.593907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.710021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 174.201984] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.806065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.957751] Lustre: lustre-OST0001: new disk, initializing [ 188.961151] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.967276] Lustre: 9500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.037106] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.096944] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.240318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.259723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.358595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.011340] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.217207] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 230.260905] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.919171] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing yml_node [ 241.629366] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 244.515881] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 247.147148] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 249.122472] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 09:31:34 EDT 2026 [ 266.161902] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 267.869559] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.690960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.713022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.723802] Lustre: Skipped 2 previous similar messages [ 286.755131] Lustre: Skipped 3 previous similar messages [ 291.657515] Lustre: server umount lustre-MDT0000 complete [ 295.216139] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800743 with bad export cookie 4716643608761611688 [ 295.220295] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.240725] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.930536] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.931076] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.945798] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 296.964196] Lustre: Skipped 1 previous similar message [ 296.965301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.978979] Lustre: Skipped 1 previous similar message [ 300.975019] LustreError: 6500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.975082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.988485] LustreError: 6500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 301.001857] Lustre: Skipped 1 previous similar message [ 301.814607] Lustre: server umount lustre-MDT0001 complete [ 305.343544] Lustre: server umount lustre-OST0000 complete [ 308.784715] Lustre: server umount lustre-OST0001 complete [ 315.043244] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 325.962460] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 363.875740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.126217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.161539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.243189] Lustre: lustre-MDT0000: new disk, initializing [ 364.306843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.321062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.884409] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 380.176397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.345771] Lustre: 15060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 380.399798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 380.405761] Lustre: Skipped 1 previous similar message [ 380.488984] Lustre: lustre-MDT0001: new disk, initializing [ 380.599645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.637726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.650212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.920896] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 395.449246] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 400.547706] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.753229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 407.029383] Lustre: lustre-OST0000: new disk, initializing [ 407.033659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 407.040760] Lustre: 17029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 407.128745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.130416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.147710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.320197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 414.444138] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 425.428299] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 430.918776] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 431.155772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 435.318916] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 435.476280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 440.298914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.309483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.315600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.336927] Lustre: Skipped 2 previous similar messages [ 443.377990] Lustre: server umount lustre-OST0000 complete [ 447.457685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 447.468784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.497516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 447.504531] Lustre: Skipped 1 previous similar message [ 455.654379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.661224] Lustre: Skipped 3 previous similar messages [ 461.283832] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.535906] Lustre: server umount lustre-MDT0000 complete [ 465.754585] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800913 with bad export cookie 4716643608761613228 [ 465.757206] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.771494] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 466.003967] Lustre: server umount lustre-MDT0001 complete [ 471.896242] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:35:17 (1786800917) [ 492.246300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.067162] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.198706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.300876] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 498.662171] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.759886] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.889590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.773397] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 520.579261] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.198928] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 530.844592] Lustre: Failing over lustre-MDT0000 [ 530.885325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.921530] Lustre: Skipped 1 previous similar message [ 530.946102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.960435] Lustre: Skipped 3 previous similar messages [ 531.353571] Lustre: server umount lustre-MDT0000 complete [ 534.065165] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.078829] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 549.349774] Lustre: 3627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786800981/real 1786800981] req@ffffa0616eb91180 x1873596123347968/t0(0) o400->MGC192.168.204.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786800997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 549.387378] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.466213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.509942] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.526370] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 560.617764] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4174e25f505a4132 [ 560.635687] Lustre: MGC192.168.204.152@tcp: Connection restored to 0@lo (at 0@lo) [ 560.844860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.852848] Lustre: Skipped 1 previous similar message [ 564.757981] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 566.245204] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 566.261552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 566.293834] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 574.134440] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 579.049301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.075669] Lustre: Skipped 1 previous similar message [ 579.088738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.095781] Lustre: Skipped 1 previous similar message [ 592.352908] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 592.644525] Lustre: server umount lustre-MDT0000 complete [ 594.404334] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.429291] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 596.951817] LustreError: 22568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801044 with bad export cookie 4716643608761614642 [ 596.960330] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.965566] LustreError: 22568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 597.336145] Lustre: server umount lustre-MDT0001 complete [ 605.327641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.971599] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 618.123540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.222198] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 631.078060] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 636.432568] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 638.487134] Lustre: Failing over lustre-MDT0000 [ 638.748739] Lustre: server umount lustre-MDT0000 complete [ 638.946129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.952151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.962463] Lustre: Skipped 1 previous similar message [ 656.354365] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786801088/real 1786801088] req@ffffa06144132300 x1873596123377664/t0(0) o400->MGC192.168.204.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786801104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 656.379177] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.195036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.474444] LustreError: 23246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.498904] LustreError: 23246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 666.902921] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 671.315733] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 672.230091] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 672.235761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 672.248362] Lustre: Skipped 1 previous similar message [ 672.264745] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 680.505488] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 684.309455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.316952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.317588] Lustre: Skipped 2 previous similar messages [ 684.325938] Lustre: Skipped 5 previous similar messages [ 690.587880] Lustre: server umount lustre-MDT0000 complete [ 693.925217] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801141 with bad export cookie 4716643608761615587 [ 693.926863] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.935570] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 694.265394] Lustre: server umount lustre-MDT0001 complete [ 702.685671] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:39:08 (1786801148) [ 719.538924] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 729.497340] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 774.510090] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 785.511072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.686898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 785.705579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 785.770215] Lustre: lustre-MDT0000: new disk, initializing [ 785.867824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.874942] Lustre: Skipped 3 previous similar messages [ 785.915329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 790.139124] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 800.491622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.617230] Lustre: 29615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 800.861087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 805.411969] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 815.845248] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 823.491749] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 831.106281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.387550] Lustre: 31583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 833.118582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 833.284019] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 833.406222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 833.418538] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 838.641982] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 848.517797] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 855.155870] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 874.470287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.481301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.484796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.484804] Lustre: Skipped 2 previous similar messages [ 874.494370] Lustre: Skipped 1 previous similar message [ 874.499367] LustreError: Skipped 1 previous similar message [ 880.503643] Lustre: server umount lustre-OST0000 complete [ 887.780093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 891.236654] Lustre: server umount lustre-MDT0000 complete [ 894.947163] LustreError: 29628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.999346] LustreError: 29628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 895.931162] LustreError: 31593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801343 with bad export cookie 4716643608761615986 [ 895.934156] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.944054] LustreError: 31593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 896.911819] Lustre: server umount lustre-MDT0001 complete [ 909.971089] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 912.752035] Key type lgssc unregistered [ 913.018537] LNet: 33565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.034315] LNetError: 33565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.055267] LNet: Removed LNI 192.168.204.152@tcp [ 914.503256] Key type .llcrypt unregistered [ 914.505075] Key type ._llcrypt unregistered [ 943.623374] Key type ._llcrypt registered [ 943.625687] Key type .llcrypt registered [ 943.725823] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 944.869795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.908121] alg: No test for adler32 (adler32-zlib) [ 946.117220] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 946.407840] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 948.152212] Key type lgssc registered [ 949.339945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 958.378634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 958.421714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.909186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.606686] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 969.198630] LustreError: 34980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.232816] LustreError: 34980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 974.314371] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.462123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.891471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.079453] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 988.112643] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 994.548971] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.424299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.909598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.995145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1009.811160] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1018.786520] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1019.337741] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1036.032721] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:44:41 (1786801481) [ 1038.153666] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1040.131465] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1048.246731] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:44:53 (1786801493) [ 1059.594244] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:45:04 (1786801504) [ 1060.873103] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1070.790581] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1070.799170] Lustre: Skipped 2 previous similar messages [ 1072.924907] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:45:18 (1786801518) [ 1075.691858] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1075.701703] Lustre: Skipped 2 previous similar messages [ 1085.151142] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1085.155745] Lustre: Skipped 4 previous similar messages [ 1099.262465] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:45:44 (1786801544) [ 1102.184288] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1105.763365] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1115.781920] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:46:00 (1786801560) [ 1117.801123] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1117.806120] Lustre: Skipped 6 previous similar messages [ 1133.792627] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:46:18 (1786801578) [ 1152.482290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1152.496412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.513437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1152.526762] Lustre: Skipped 1 previous similar message [ 1152.554807] Lustre: Skipped 1 previous similar message [ 1156.800899] Lustre: server umount lustre-OST0000 complete [ 1162.721748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.728530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.757928] Lustre: Skipped 1 previous similar message [ 1167.490986] Lustre: server umount lustre-MDT0000 complete [ 1167.842452] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.873558] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1171.777954] LustreError: 37903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801619 with bad export cookie 8753499580393759228 [ 1171.792764] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.793112] LustreError: 37903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1172.292573] Lustre: server umount lustre-MDT0001 complete [ 1183.032073] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1186.552926] Key type lgssc unregistered [ 1186.940896] LNet: 42836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.951176] LNetError: 42836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.973995] LNet: Removed LNI 192.168.204.152@tcp [ 1188.191524] Key type .llcrypt unregistered [ 1188.199031] Key type ._llcrypt unregistered [ 1208.833877] Key type ._llcrypt registered [ 1208.836501] Key type .llcrypt registered [ 1208.928919] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 1227.142558] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1228.516701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.581418] alg: No test for adler32 (adler32-zlib) [ 1229.694156] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1230.046456] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1231.800201] Key type lgssc registered [ 1232.907380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1270.151322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1270.187051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.532978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1271.565436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1271.670416] Lustre: lustre-MDT0000: new disk, initializing [ 1271.730449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.744031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1276.665749] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1286.408312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.499424] Lustre: 45991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1286.542829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1286.546654] Lustre: Skipped 1 previous similar message [ 1286.607301] Lustre: lustre-MDT0001: new disk, initializing [ 1286.694761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.738708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1286.754234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1291.673232] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1300.998049] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1307.479820] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1316.023150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.364702] Lustre: lustre-OST0000: new disk, initializing [ 1316.368930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1316.378823] Lustre: 47962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1316.481740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.091766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1322.106802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1322.248509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1323.897787] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1335.336368] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1341.895657] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1342.348435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1347.245582] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1347.508334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1353.186980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.189194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.204044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.206491] LustreError: Skipped 1 previous similar message [ 1353.251829] Lustre: Skipped 1 previous similar message [ 1355.888293] Lustre: server umount lustre-OST0000 complete [ 1363.436314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.441501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1363.457826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1363.457836] Lustre: Skipped 1 previous similar message [ 1363.468780] Lustre: Skipped 1 previous similar message [ 1365.988931] Lustre: server umount lustre-MDT0000 complete [ 1369.776708] LustreError: 45983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801817 with bad export cookie 11242451712284438552 [ 1369.789669] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1369.795736] LustreError: 45983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1370.031837] Lustre: server umount lustre-MDT0001 complete [ 1380.503824] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:50:25 (1786801825) [ 1386.945527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.476463] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.499985] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1387.590323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.216447] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1392.611692] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.735721] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.124815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.494822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1405.332405] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1411.968354] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1417.401865] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1423.796890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.185209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.079384] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1437.651065] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.317271] LustreError: 52952:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1443.090343] Lustre: Modifying parameter general.timeout=1 in log params [ 1444.677778] Lustre: Modifying parameter general.timeout=3 in log params [ 1444.683643] Lustre: Skipped 1 previous similar message [ 1447.059985] Lustre: Modifying parameter general.timeout=6 in log params [ 1447.062858] Lustre: Skipped 2 previous similar messages [ 1451.221755] Lustre: Modifying parameter general.timeout=11 in log params [ 1451.226461] Lustre: Skipped 4 previous similar messages [ 1459.638043] Lustre: Modifying parameter general.timeout=20 in log params [ 1459.642371] Lustre: Skipped 8 previous similar messages [ 1475.785926] Lustre: Modifying parameter general.timeout=41 in log params [ 1475.793300] Lustre: Skipped 20 previous similar messages [ 1508.178745] Lustre: Modifying parameter general.timeout=75 in log params [ 1508.182101] Lustre: Skipped 33 previous similar messages [ 1565.386796] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:53:30 (1786802010) [ 1568.217475] Lustre: 59892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1569.768870] Lustre: 59990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1572.553751] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1572.560469] Lustre: Skipped 59 previous similar messages [ 1573.376790] Lustre: 60203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1573.386803] Lustre: 60203:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1611.493859] Lustre: 61768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1611.505959] Lustre: 61768:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1611.531110] LustreError: 61768:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1617.633347] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:54:21 (1786802061) [ 1635.821614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.828502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.832025] Lustre: Skipped 2 previous similar messages [ 1635.852941] Lustre: Skipped 3 previous similar messages [ 1638.287745] Lustre: server umount lustre-MDT0000 complete [ 1640.930241] LustreError: 51150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.957987] LustreError: 51150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1641.952762] LustreError: 50403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802089 with bad export cookie 11242451712284439434 [ 1641.960341] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1641.965928] LustreError: 50403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1642.385385] Lustre: server umount lustre-MDT0001 complete [ 1656.789917] Lustre: server umount lustre-OST0000 complete [ 1682.354749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.695111] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1682.915282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.514916] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1696.094143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.164391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1696.185948] Lustre: 64302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1701.354280] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1708.367464] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1716.162195] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1725.717749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.173121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1726.915101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.933692] Lustre: Skipped 1 previous similar message [ 1736.219981] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1745.059918] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1747.087852] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1747.094599] Lustre: Skipped 18 previous similar messages [ 1747.138871] Lustre: 66869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1758.179367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.189692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.199306] Lustre: Skipped 1 previous similar message [ 1760.412137] Lustre: server umount lustre-OST0000 complete [ 1767.906641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1767.912787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.924021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.926444] Lustre: Skipped 1 previous similar message [ 1770.326989] Lustre: server umount lustre-MDT0000 complete [ 1773.544253] LustreError: 64312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.564180] LustreError: 64312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1773.905562] LustreError: 64293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802221 with bad export cookie 11242451712284442535 [ 1773.914274] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.928084] LustreError: 64293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1774.226197] Lustre: server umount lustre-MDT0001 complete [ 1784.705363] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1787.047236] Key type lgssc unregistered [ 1787.299892] LNet: 68075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.306822] LNetError: 68075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.320194] LNet: Removed LNI 192.168.204.152@tcp [ 1788.025547] Key type .llcrypt unregistered [ 1788.028682] Key type ._llcrypt unregistered [ 1813.411449] Key type ._llcrypt registered [ 1813.413361] Key type .llcrypt registered [ 1813.494866] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1814.777670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.815483] alg: No test for adler32 (adler32-zlib) [ 1815.811856] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1816.016863] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1817.760248] Key type lgssc registered [ 1818.938551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.012739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.027744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.470577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.763695] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1839.077055] LustreError: 69488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.108544] LustreError: 69488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1841.788225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.327256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1846.287848] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1854.303828] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.775871] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1869.442053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1870.161470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.707402] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1884.651795] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1893.815505] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1903.586511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.606991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.627559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.173352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.175252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.197144] Lustre: Skipped 1 previous similar message [ 1907.203072] Lustre: Skipped 1 previous similar message [ 1908.997744] Lustre: server umount lustre-MDT0000 complete [ 1912.292607] LustreError: 70219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.311140] LustreError: 70219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1912.876875] LustreError: 72172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802360 with bad export cookie 18414848270739022615 [ 1912.886963] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1912.894183] LustreError: 72172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1913.583144] Lustre: server umount lustre-MDT0001 complete [ 1928.254689] Lustre: server umount lustre-OST0000 complete [ 1954.535952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.805940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1954.841888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1955.142327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.157684] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1968.280230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.448683] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1968.490663] Lustre: 74364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1969.058223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.743450] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1983.741235] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1990.886131] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1997.664236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1997.882057] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1998.295991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.960194] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2009.868548] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2011.581610] Lustre: 76938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2013.136902] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2020.321731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2020.331722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.344790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.350511] Lustre: Skipped 1 previous similar message [ 2024.795146] Lustre: server umount lustre-OST0000 complete [ 2030.561369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2030.576241] LustreError: Skipped 1 previous similar message [ 2030.583039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.609169] Lustre: Skipped 1 previous similar message [ 2030.626483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2030.635339] Lustre: Skipped 1 previous similar message [ 2035.213567] Lustre: server umount lustre-MDT0000 complete [ 2036.198677] LustreError: 74375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.214367] LustreError: 74375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2040.290177] LustreError: 74375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.303933] LustreError: 74375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2040.336535] LustreError: 74356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802488 with bad export cookie 18414848270739023462 [ 2040.346789] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.347882] LustreError: 74356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2040.856624] Lustre: server umount lustre-MDT0001 complete [ 2053.322050] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2056.250487] Key type lgssc unregistered [ 2056.519750] LNet: 78163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.530826] LNetError: 78163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.552870] LNet: Removed LNI 192.168.204.152@tcp [ 2057.217675] Key type .llcrypt unregistered [ 2057.220433] Key type ._llcrypt unregistered [ 2083.388956] Key type ._llcrypt registered [ 2083.393195] Key type .llcrypt registered [ 2083.513558] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2084.606519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.694603] alg: No test for adler32 (adler32-zlib) [ 2085.823615] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2086.077168] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2087.721511] Key type lgssc registered [ 2088.917905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.833429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2095.855714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.460052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.908590] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2107.883146] LustreError: 79578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.925572] LustreError: 79578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2110.925189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.474149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.392958] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2124.071556] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2130.021638] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2137.397227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2137.856654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.231243] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2150.777308] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2159.524418] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2160.318911] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2162.083190] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2162.093503] Lustre: Skipped 1 previous similar message [ 2164.812548] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2164.825974] Lustre: Skipped 2 previous similar messages [ 2168.876492] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2168.879749] Lustre: Skipped 4 previous similar messages [ 2176.898943] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.906433] Lustre: Skipped 9 previous similar messages [ 2193.801760] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2193.810993] Lustre: Skipped 18 previous similar messages [ 2226.101548] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.111322] Lustre: Skipped 30 previous similar messages [ 2265.151475] Lustre: 87678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2268.267556] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:05:13 (1786802713) [ 2269.979899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2271.913216] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:05:17 (1786802717) [ 2279.400628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.412414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.429181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.449679] Lustre: Skipped 1 previous similar message [ 2283.019075] Lustre: server umount lustre-OST0000 complete [ 2289.634627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.639094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.643424] Lustre: Skipped 2 previous similar messages [ 2289.653189] Lustre: Skipped 1 previous similar message [ 2293.386767] Lustre: server umount lustre-MDT0000 complete [ 2294.754936] LustreError: 79577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.782530] LustreError: 79577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2298.162283] LustreError: 79559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802746 with bad export cookie 11874181475767881925 [ 2298.173429] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.177141] LustreError: 79559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2298.780730] Lustre: server umount lustre-MDT0001 complete [ 2311.882938] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2314.659602] Key type lgssc unregistered [ 2315.053601] LNet: 88929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.063155] LNetError: 88929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.081424] LNet: Removed LNI 192.168.204.152@tcp [ 2316.179723] Key type .llcrypt unregistered [ 2316.182966] Key type ._llcrypt unregistered [ 2320.717029] Key type ._llcrypt registered [ 2320.727892] Key type .llcrypt registered [ 2320.846701] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2325.734380] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules [ 2326.721463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2326.819655] alg: No test for adler32 (adler32-zlib) [ 2327.968906] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2328.430976] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2330.210344] Key type lgssc registered [ 2331.366562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.556847] LustreError: 90135:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2348.782915] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2354.302048] LustreError: 90135:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2358.902140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2358.946576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.488169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.035492] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2371.043395] LustreError: 91156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.068739] LustreError: 91156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2376.124750] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:07:01 (1786802821) [ 2376.168988] LustreError: 91157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.292344] LustreError: 91156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.746359] LustreError: 91188:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2381.760038] LustreError: 91188:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2381.774796] LustreError: 91188:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2382.424226] Lustre: server umount lustre-MDT0000 complete [ 2398.368106] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2402.546113] Key type lgssc unregistered [ 2403.049481] LNet: 92684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.076075] LNetError: 92684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.115907] LNet: Removed LNI 192.168.204.152@tcp [ 2404.240113] Key type .llcrypt unregistered [ 2404.251892] Key type ._llcrypt unregistered [ 2426.365340] Key type ._llcrypt registered [ 2426.367613] Key type .llcrypt registered [ 2426.470629] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2427.713201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.826677] alg: No test for adler32 (adler32-zlib) [ 2428.966502] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2429.288852] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2430.960755] Key type lgssc registered [ 2432.384263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.748866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.765058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.280308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.975632] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2454.505637] LustreError: 93902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.552406] LustreError: 93902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2459.622262] LustreError: 93903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.516804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.073907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2468.417968] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2477.006581] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2483.042680] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2489.723615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.838571] LustreError: 95732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.488375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.493840] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2503.308942] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.182768] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:10:01 (1786803001) [ 2557.856859] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2559.939666] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:10:05 (1786803005) [ 2565.089903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.097445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.110218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.674374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.689368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.699758] Lustre: Skipped 1 previous similar message [ 2568.717154] Lustre: Skipped 2 previous similar messages [ 2570.553201] Lustre: server umount lustre-MDT0000 complete [ 2573.813094] LustreError: 93898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.831228] LustreError: 93898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2575.088752] LustreError: 93884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803022 with bad export cookie 16659590556126464120 [ 2575.106849] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.671852] Lustre: server umount lustre-MDT0001 complete [ 2589.434679] Lustre: server umount lustre-OST0000 complete [ 2603.200234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.917332] LustreError: 98177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.096612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.509543] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2613.221953] LustreError: 98178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.243169] LustreError: 98178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2621.417313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.171888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.161926] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2635.658906] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2641.796413] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2656.497131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2656.711654] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2656.726309] LustreError: 98164:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2656.772497] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2656.785139] LustreError: 100261:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2656.795760] LustreError: 100261:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2656.807108] LustreError: 100261:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2656.874590] Lustre: server umount lustre-OST0000 complete [ 2656.878643] LustreError: 100261:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2665.453060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2665.637840] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2665.643619] LustreError: 98167:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2665.657753] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2665.663645] LustreError: 100726:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2665.668705] LustreError: 100726:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2665.673607] LustreError: 100726:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2665.725191] Lustre: server umount lustre-OST0000 complete [ 2665.728481] LustreError: 100726:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2675.351599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2675.570091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2675.576633] Lustre: Found index 0 for lustre-OST0000, updating log [ 2675.581830] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2675.691901] Lustre: lustre-OST0000: new disk, initializing [ 2675.698268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2675.711182] Lustre: 101262:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2676.098035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.599601] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2685.471580] Lustre: Failing over lustre-OST0000 [ 2685.806193] Lustre: server umount lustre-OST0000 complete [ 2691.557861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.565038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.572484] Lustre: Skipped 1 previous similar message [ 2691.587725] Lustre: Skipped 1 previous similar message [ 2695.980850] Lustre: server umount lustre-MDT0000 complete [ 2697.698365] LustreError: 98177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.722301] LustreError: 98177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2700.358210] LustreError: 101261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803148 with bad export cookie 16659590556126466864 [ 2700.368142] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.375064] LustreError: 101261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2700.711156] Lustre: server umount lustre-MDT0001 complete [ 2710.243134] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:12:35 (1786803155) [ 2728.494564] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2738.896546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.283326] LustreError: 103607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.402099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.995983] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2753.261923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.456945] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2761.429586] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2767.829549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2774.013673] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2782.203806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.298748] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2782.391398] Lustre: lustre-OST0001: new disk, initializing [ 2782.395658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2782.403635] Lustre: 105926:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2782.708358] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.718131] Lustre: Skipped 2 previous similar messages [ 2788.731601] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2790.531896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2790.545421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2790.703555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2796.798755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2802.857274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2810.337840] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.353991] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.385342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2810.391292] Lustre: Skipped 2 previous similar messages [ 2814.008082] Lustre: server umount lustre-MDT0001 complete [ 2816.493189] LustreError: 103602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.506932] LustreError: 103602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2825.541836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.696299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2825.709721] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2825.781982] Lustre: 103596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2827.690241] LustreError: 107462:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2827.706876] LustreError: 107462:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2827.725987] LustreError: 103599:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2827.891436] LustreError: 107470:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.152@tcp: cfg command failed: rc = -17 [ 2827.918916] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2827.918916] [ 2832.368344] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2833.048936] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2841.959047] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:14:47 (1786803287) [ 2857.854983] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2875.879073] Lustre: 109539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2903.096901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2906.676618] Lustre: 110676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2999.265577] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.267388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2999.286312] Lustre: Skipped 4 previous similar messages [ 2999.305440] Lustre: Skipped 7 previous similar messages [ 3002.192411] Lustre: server umount lustre-MDT0000 complete [ 3004.387268] LustreError: 108062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.404048] LustreError: 108062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3005.901553] LustreError: 105925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803453 with bad export cookie 16659590556126467858 [ 3005.910670] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.917082] LustreError: 105925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3006.407932] Lustre: server umount lustre-MDT0001 complete [ 3021.183637] Lustre: server umount lustre-OST0000 complete [ 3035.192700] Lustre: server umount lustre-OST0001 complete [ 3041.066693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3050.981188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3059.144401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3067.861736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3083.228505] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3092.921909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.955399] Lustre: lustre-MDT0000: reset Object Index mappings [ 3093.534168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.549148] Lustre: Skipped 1 previous similar message [ 3098.595531] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3109.275981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.295140] Lustre: lustre-MDT0001: reset Object Index mappings [ 3114.606976] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3119.036648] Lustre: 115778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3126.138833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.631397] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3134.989965] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3139.124091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3141.331690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3147.276569] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3147.359267] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3149.483167] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3157.273258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3161.260205] Lustre: 117653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3216.225703] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:21:00 (1786803660) [ 3242.983690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3242.996411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.021452] Lustre: Skipped 1 previous similar message [ 3243.028585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.883547] Lustre: server umount lustre-MDT0000 complete [ 3249.246455] LustreError: 114610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803697 with bad export cookie 16659590556126514730 [ 3249.266783] LustreError: 114610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3249.276856] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.970311] Lustre: server umount lustre-MDT0001 complete [ 3266.250941] Lustre: server umount lustre-OST0000 complete [ 3276.327371] Lustre: server umount lustre-OST0001 complete [ 3282.903755] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 3292.179981] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3331.638210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.874871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3331.909668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3332.052386] Lustre: lustre-MDT0000: new disk, initializing [ 3332.136760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.143683] Lustre: Skipped 3 previous similar messages [ 3332.171325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3336.939834] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3344.126961] Lustre: server umount lustre-MDT0000 complete [ 3353.181304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.519622] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3353.533933] Lustre: 121922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3353.553791] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3358.695631] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3364.941901] Lustre: server umount lustre-MDT0000 complete [ 3373.924433] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:23:39 (1786803819) [ 3375.629211] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3377.380349] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:23:42 (1786803822) [ 3396.377487] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3406.389958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.371464] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3418.723317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.853667] Lustre: 123972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3418.911093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3418.915287] Lustre: Skipped 1 previous similar message [ 3418.984442] Lustre: lustre-MDT0001: new disk, initializing [ 3419.094879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3419.107415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3423.343377] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3427.863350] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3434.189438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.430454] Lustre: lustre-OST0000: new disk, initializing [ 3434.435924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3434.442936] Lustre: 125549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3436.335609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3436.342782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3436.474926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3440.239667] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3451.013266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3451.258736] Lustre: lustre-OST0001: new disk, initializing [ 3451.264175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3451.270701] Lustre: 126420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3453.213893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3453.222852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3453.385645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3457.781701] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3466.911339] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3470.956273] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3862.130448] Lustre: 132833:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3862.149459] Lustre: 132833:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3862.159461] Lustre: 132833:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3862.174520] Lustre: 132833:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3862.183092] Lustre: 132833:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3862.189149] Lustre: 132833:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4171.376358] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:36:56 (1786804616) [ 4187.106076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4187.114506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.121803] Lustre: Skipped 5 previous similar messages [ 4187.125455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.130946] Lustre: Skipped 8 previous similar messages [ 4190.691448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4190.693359] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.720855] Lustre: Skipped 1 previous similar message [ 4192.356698] Lustre: server umount lustre-MDT0000 complete [ 4195.829325] LustreError: 127239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.846617] LustreError: 127239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 4196.202736] LustreError: 123962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804644 with bad export cookie 16659590556126583540 [ 4196.212729] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.222642] LustreError: 123962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4200.930994] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.946512] Lustre: Skipped 1 previous similar message [ 4200.950801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4200.959078] Lustre: Skipped 4 previous similar messages [ 4202.841146] Lustre: server umount lustre-MDT0001 complete [ 4213.003780] Lustre: server umount lustre-OST0000 complete [ 4223.303148] Lustre: server umount lustre-OST0001 complete [ 4230.406984] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 4242.454329] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4282.250069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.730671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4282.774798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4282.926532] Lustre: lustre-MDT0000: new disk, initializing [ 4283.047625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.055581] Lustre: Skipped 5 previous similar messages [ 4283.077192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4287.966860] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4298.743215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.881219] Lustre: 135897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4298.938128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4298.942517] Lustre: Skipped 1 previous similar message [ 4299.036499] Lustre: lustre-MDT0001: new disk, initializing [ 4299.253228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4299.266220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4303.753863] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4313.048344] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4318.964712] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4326.756084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.069960] Lustre: lustre-OST0000: new disk, initializing [ 4327.074074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4327.081755] Lustre: 137869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4327.160584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.164503] Lustre: Skipped 1 previous similar message [ 4328.421758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4328.446794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4328.619889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4333.543594] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4342.963724] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4352.866593] Lustre: lustre-MDD0000: changelog on [ 4355.625968] Lustre: lustre-MDD0001: changelog on [ 6465.257115] Lustre: lustre-MDD0001: changelog off [ 6467.031231] Lustre: 139023:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6467.920923] Lustre: lustre-MDD0000: changelog off [ 6473.697987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6473.706535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.719478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6473.719489] Lustre: Skipped 2 previous similar messages [ 6473.751184] Lustre: Skipped 1 previous similar message [ 6478.827352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.832019] Lustre: Skipped 2 previous similar messages [ 6479.053668] Lustre: server umount lustre-OST0000 complete [ 6483.938390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.959670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6483.968029] Lustre: Skipped 1 previous similar message [ 6489.063367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.067643] Lustre: Skipped 1 previous similar message [ 6489.728870] Lustre: server umount lustre-MDT0000 complete [ 6493.476062] LustreError: 135890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806941 with bad export cookie 16659590556127404269 [ 6493.486753] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6493.757787] Lustre: server umount lustre-MDT0001 complete [ 6505.000982] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6507.819934] Key type lgssc unregistered [ 6508.133935] LNet: 147032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6508.171077] LNetError: 147032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6509.225474] LNet: Removed LNI 192.168.204.152@tcp [ 6510.440189] Key type .llcrypt unregistered [ 6510.448975] Key type ._llcrypt unregistered [ 6532.896348] Key type ._llcrypt registered [ 6532.899053] Key type .llcrypt registered [ 6533.001374] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 6550.676782] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6551.655791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6551.766091] alg: No test for adler32 (adler32-zlib) [ 6552.876939] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6553.115681] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6554.808297] Key type lgssc registered [ 6556.485168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6594.266400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6594.311213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.757111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6595.843727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6595.966408] Lustre: lustre-MDT0000: new disk, initializing [ 6596.055521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.074520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6600.941742] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6613.956052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.221902] Lustre: 150239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6614.307919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6614.311975] Lustre: Skipped 1 previous similar message [ 6614.379913] Lustre: lustre-MDT0001: new disk, initializing [ 6614.491439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6614.556738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6614.579826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6619.891189] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6631.201357] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6637.652188] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6644.992038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.448685] Lustre: lustre-OST0000: new disk, initializing [ 6645.460599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6645.471882] Lustre: 152209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6645.602811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.396444] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6655.034899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6655.041345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6655.130341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6660.779490] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6665.654816] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_module obdecho/obdecho [ 6667.598574] LustreError: 153212:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6667.604832] LustreError: 153212:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6669.284056] LustreError: 153308:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6681.059092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6681.060964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.062572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.094760] Lustre: Skipped 2 previous similar messages [ 6686.183680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6686.188733] Lustre: Skipped 2 previous similar messages [ 6687.239603] Lustre: server umount lustre-MDT0000 complete [ 6690.705986] LustreError: 150230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807138 with bad export cookie 10562084080068651322 [ 6690.710388] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.720368] LustreError: 150230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6691.048292] Lustre: server umount lustre-MDT0001 complete [ 6705.186122] Lustre: server umount lustre-OST0000 complete [ 6726.141734] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 6735.929816] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6776.729683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.964616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6776.998143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6777.086534] Lustre: lustre-MDT0000: new disk, initializing [ 6777.151538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.183506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6782.022577] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6795.198082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.377932] Lustre: 157025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6795.422869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6795.426150] Lustre: Skipped 1 previous similar message [ 6795.498976] Lustre: lustre-MDT0001: new disk, initializing [ 6795.623519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6795.728865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6795.751357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6801.660331] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6813.403622] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6819.952342] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6827.755369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.978290] Lustre: lustre-OST0000: new disk, initializing [ 6827.982205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6827.988800] Lustre: 158995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6828.074163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.741194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6833.765648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6833.900250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6835.218566] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6846.602167] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.759795] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6850.994255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6856.098274] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6856.388708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6859.747506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6859.748038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.771477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6859.771487] Lustre: Skipped 2 previous similar messages [ 6859.772171] LustreError: Skipped 1 previous similar message [ 6859.800967] Lustre: Skipped 1 previous similar message [ 6864.638130] Lustre: server umount lustre-OST0000 complete [ 6869.988272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6869.997900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6870.018098] Lustre: Skipped 1 previous similar message [ 6870.031849] Lustre: Skipped 1 previous similar message [ 6874.589842] Lustre: server umount lustre-MDT0000 complete [ 6878.699405] LustreError: 157819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807326 with bad export cookie 10562084080068652281 [ 6878.709626] LustreError: 157819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6878.716574] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.178102] Lustre: server umount lustre-MDT0001 complete [ 6883.076466] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:22:08 (1786807328) [ 6885.053324] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6886.723879] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:22:12 (1786807332) [ 6895.268826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.866473] LustreError: 161408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6895.885551] LustreError: 161408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6895.968078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.967883] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6901.261308] LustreError: 161409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.389509] LustreError: 161408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.354942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6914.396289] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6923.085370] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6929.778770] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6938.540038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.887195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.895280] Lustre: Skipped 1 previous similar message [ 6946.459561] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6953.616793] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6964.897935] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [ 6964.905471] Lustre: Skipped 1 previous similar message [ 6965.737788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.755816] Lustre: Skipped 2 previous similar messages [ 6966.123644] Lustre: server umount lustre-MDT0000 complete [ 6969.360045] LustreError: 161389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807417 with bad export cookie 10562084080068653163 [ 6969.375406] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.378335] LustreError: 161389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6970.012593] LustreError: 163899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6970.853089] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.882055] LustreError: 161403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.882953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6974.907110] LustreError: 161403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6974.913838] Lustre: Skipped 5 previous similar messages [ 6975.834824] Lustre: server umount lustre-MDT0001 complete [ 6983.089920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6995.459258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.781965] LustreError: 165248:0:(llog.c:1655:llog_backup()) MGC192.168.204.152@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6995.799672] Lustre: 165248:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.152@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6999.528665] LustreError: 165253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.554135] LustreError: 165253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7016.078823] LustreError: 165255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.098369] LustreError: 165255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7049.699100] LustreError: 165255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.708987] LustreError: 165255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7056.352502] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7056.359904] Lustre: 165248:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7097.346513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7101.649489] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7102.448603] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7109.500259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.726096] LustreError: 165274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7114.751799] LustreError: 165274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7118.571247] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7120.364310] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7125.466613] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7126.854614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7131.671078] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7133.277654] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7151.073288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7151.079837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.081994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.104143] Lustre: Skipped 2 previous similar messages [ 7151.113045] Lustre: Skipped 3 previous similar messages [ 7156.946213] Lustre: server umount lustre-MDT0000 complete [ 7160.659548] LustreError: 161390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807608 with bad export cookie 10562084080068659617 [ 7160.676491] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7160.678457] LustreError: 161390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7161.143771] Lustre: server umount lustre-MDT0001 complete [ 7174.359758] Lustre: server umount lustre-OST0000 complete [ 7197.214222] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 7208.167679] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7256.212664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.430253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7256.454822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7256.530453] Lustre: lustre-MDT0000: new disk, initializing [ 7256.591772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.596545] Lustre: Skipped 1 previous similar message [ 7256.609619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7262.129433] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7274.636840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.917140] Lustre: 170354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.109408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7282.499912] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7295.849831] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7304.173435] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7312.080635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7312.242911] Lustre: lustre-OST0000: new disk, initializing [ 7312.247487] Lustre: Skipped 1 previous similar message [ 7312.250430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7312.253986] Lustre: Skipped 2 previous similar messages [ 7312.258501] Lustre: 172324:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7317.596564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7317.613581] Lustre: Skipped 1 previous similar message [ 7317.623254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7317.748606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7318.407370] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7328.904093] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7335.914674] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7336.348585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7341.286568] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7341.499617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7343.589798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.596868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.597844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7343.608517] Lustre: Skipped 1 previous similar message [ 7343.642906] Lustre: Skipped 4 previous similar messages [ 7349.601688] Lustre: server umount lustre-OST0000 complete [ 7359.901824] Lustre: server umount lustre-MDT0000 complete [ 7363.314964] LustreError: 170348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807811 with bad export cookie 10562084080068676452 [ 7363.327797] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.638722] Lustre: server umount lustre-MDT0001 complete [ 7371.822838] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:30:17 (1786807817) [ 7378.216560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.680632] LustreError: 174735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7383.501550] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7392.928193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.305792] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7408.075405] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7415.409153] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7422.877835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.698345] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7436.261418] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7449.231958] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:31:34 (1786807894) [ 7454.181653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7454.185552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.199124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7454.203337] Lustre: Skipped 3 previous similar messages [ 7454.217034] Lustre: Skipped 7 previous similar messages [ 7459.206635] Lustre: server umount lustre-OST0000 complete [ 7468.924048] Lustre: server umount lustre-MDT0000 complete [ 7473.070533] LustreError: 174716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807920 with bad export cookie 10562084080068677334 [ 7473.088435] LustreError: 174716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7473.104939] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7473.497640] Lustre: server umount lustre-MDT0001 complete [ 7488.399883] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7491.053617] Key type lgssc unregistered [ 7491.347819] LNet: 178465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7491.354709] LNetError: 178465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.394548] LNet: Removed LNI 192.168.204.152@tcp [ 7493.342491] Key type .llcrypt unregistered [ 7493.346355] Key type ._llcrypt unregistered [ 7514.409961] Key type ._llcrypt registered [ 7514.412349] Key type .llcrypt registered [ 7514.494966] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7515.746554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7515.799880] alg: No test for adler32 (adler32-zlib) [ 7516.948990] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 7517.231781] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7518.944208] Key type lgssc registered [ 7519.961222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7528.827716] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7528.849179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.490691] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7549.664336] LustreError: 179710:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 7554.848542] LustreError: 179710:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7554.862270] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7554.869218] LustreError: MGC192.168.204.152@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7554.890396] LustreError: 179710:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7554.902086] LustreError: 179710:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7554.909302] LustreError: 179710:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7555.001923] Lustre: server umount lustre-OST0000 complete [ 7555.010750] LustreError: 179710:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7565.192877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.838645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.472804] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7577.059627] LustreError: 180212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7577.085406] LustreError: 180212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7581.173052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.619449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.114307] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7594.213883] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7600.038289] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7608.596612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.913969] LustreError: 182028:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7608.930513] Lustre: 182028:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.152@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7609.155682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.808957] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7623.936170] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7627.752695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7627.761937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.778844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.852215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.857493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.904725] Lustre: Skipped 1 previous similar message [ 7633.468602] Lustre: server umount lustre-MDT0000 complete [ 7636.966075] LustreError: 180940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.993646] LustreError: 180940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7637.523544] LustreError: 180191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808085 with bad export cookie 14886657435401002240 [ 7637.527635] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7637.539129] LustreError: 180191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7638.088602] LustreError: 180942:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000072771859 x1873603948737792/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7638.296481] Lustre: server umount lustre-MDT0001 complete [ 7652.329391] Lustre: server umount lustre-OST0000 complete [ 7661.343806] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 7671.817499] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7710.382229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7710.611761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7710.633617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7710.696754] Lustre: lustre-MDT0000: new disk, initializing [ 7710.755433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.769454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7715.015378] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7725.632397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7725.788748] Lustre: 185551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7725.883434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7725.890983] Lustre: Skipped 1 previous similar message [ 7726.004579] Lustre: lustre-MDT0001: new disk, initializing [ 7726.103282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7726.159790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7726.172703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7731.760656] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7740.603253] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7746.233438] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7752.843374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7753.029358] Lustre: lustre-OST0000: new disk, initializing [ 7753.034223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7753.039695] Lustre: 187520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7753.144443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.533423] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7762.051730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7762.074389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7762.144403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7767.495473] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7772.088330] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7772.247738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7777.296246] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7777.503557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7782.885039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.887360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7782.906026] LustreError: Skipped 1 previous similar message [ 7782.913798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7782.923959] Lustre: Skipped 3 previous similar messages [ 7785.705080] Lustre: server umount lustre-OST0000 complete [ 7792.617779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7792.623683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.636872] Lustre: Skipped 1 previous similar message [ 7792.647046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.243412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.250432] Lustre: Skipped 3 previous similar messages [ 7804.387670] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7804.577317] Lustre: server umount lustre-MDT0000 complete [ 7807.465362] LustreError: 185563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7807.489086] LustreError: 185563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7809.096861] LustreError: 187530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808256 with bad export cookie 14886657435401002961 [ 7809.123951] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7809.130546] LustreError: 187530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7809.679768] Lustre: server umount lustre-MDT0001 complete [ 7823.699022] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:37:47 (1786808267) [ 7849.102418] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7852.044577] Key type lgssc unregistered [ 7852.304697] LNet: 190469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7852.324116] LNetError: 190469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7852.341101] LNet: Removed LNI 192.168.204.152@tcp [ 7853.434658] Key type .llcrypt unregistered [ 7853.438934] Key type ._llcrypt unregistered [ 7871.282169] Key type ._llcrypt registered [ 7871.284695] Key type .llcrypt registered [ 7871.357855] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7872.231301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7872.294316] alg: No test for adler32 (adler32-zlib) [ 7873.281307] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 7873.735575] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7875.496832] Key type lgssc registered [ 7877.221737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7887.551823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7887.572640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.223066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.408417] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7899.620039] LustreError: 191690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7899.647523] LustreError: 191690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7904.741715] LustreError: 191689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.429093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.958488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7912.835275] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7922.485750] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7928.964962] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7938.818501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7939.267832] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7939.272687] LustreError: 193556:0:(llog.c:1696:llog_backup()) MGC192.168.204.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7939.292063] Lustre: 193556:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.152@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7939.496624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.908075] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7955.200969] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7957.987257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7957.998531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.008688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7962.084980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7962.090532] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.915484] Lustre: server umount lustre-MDT0000 complete [ 7967.206095] LustreError: 191690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7967.243605] LustreError: 191690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7969.082828] LustreError: 191669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808416 with bad export cookie 11039062416413884224 [ 7969.089915] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7969.093051] LustreError: 191669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7969.390932] Lustre: server umount lustre-MDT0001 complete [ 7985.316572] Lustre: server umount lustre-OST0000 complete [ 7996.244807] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 8008.692669] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8053.599443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.935908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8053.959540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8054.028081] Lustre: lustre-MDT0000: new disk, initializing [ 8054.105244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.122463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8059.190979] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8070.684363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.869198] Lustre: 197084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8070.947322] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8070.959265] Lustre: Skipped 1 previous similar message [ 8070.969107] LustreError: 197865:0:(llog.c:1696:llog_backup()) MGC192.168.204.152@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8070.992548] LustreError: 197865:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8070.999051] Lustre: 197865:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.152@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8071.017132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8071.027823] Lustre: Skipped 1 previous similar message [ 8071.185667] Lustre: lustre-MDT0001: new disk, initializing [ 8071.343379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8071.411276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8071.431162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8076.579728] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8086.558773] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8091.870671] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8099.618571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8099.828993] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8099.832993] Lustre: Skipped 1 previous similar message [ 8099.835495] LustreError: 199026:0:(llog.c:1696:llog_backup()) MGC192.168.204.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8099.842434] LustreError: 199026:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8099.847587] Lustre: 199026:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.152@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8099.855504] Lustre: 199026:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8099.872830] Lustre: lustre-OST0000: new disk, initializing [ 8099.876818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8099.882534] Lustre: 199050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8099.971559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.533965] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8109.104644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8109.115363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8109.215174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8115.781594] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8120.335372] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8120.518474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8125.417628] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8125.663426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8129.507437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8129.529065] Lustre: Skipped 1 previous similar message [ 8129.541879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8129.548445] Lustre: Skipped 2 previous similar messages [ 8133.619802] Lustre: server umount lustre-OST0000 complete [ 8138.212516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8138.233788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.250132] Lustre: Skipped 1 previous similar message [ 8138.262497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8143.398894] Lustre: server umount lustre-MDT0000 complete [ 8144.868823] LustreError: 197095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.895626] LustreError: 197095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8146.869601] LustreError: 197075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808594 with bad export cookie 11039062416413884938 [ 8146.873960] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8146.880160] LustreError: 197075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8147.199311] Lustre: server umount lustre-MDT0001 complete [ 8156.586549] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:43:21 (1786808601) [ 8176.620419] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8191.733170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.363969] LustreError: 202179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8192.424651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8198.042604] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8202.722696] LustreError: 202180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.752072] LustreError: 202180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8208.244965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.799439] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8215.545560] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8221.409675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.903181] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8235.377442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8235.536269] Lustre: lustre-OST0001: new disk, initializing [ 8235.539258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8235.544930] Lustre: 204486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8235.626233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.638116] Lustre: Skipped 2 previous similar messages [ 8241.334362] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8243.316796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8243.328802] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8243.443749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8253.058565] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8261.862766] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8264.986100] Lustre: DEBUG MARKER: ADD OST3 [ 8271.222490] loop: module loaded [ 8271.266852] loop0: detected capacity change from 0 to 204800000 [ 8271.401321] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8271.426389] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8271.437187] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8277.161412] loop0: detected capacity change from 0 to 204800000 [ 8278.485641] Lustre: DEBUG MARKER: STOP OST3 [ 8283.374216] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8283.704984] Lustre: lustre-OST0003: new disk, initializing [ 8283.709641] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8283.724303] Lustre: 206476:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8285.437918] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8285.562715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8285.568606] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8285.635872] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8287.457084] LustreError: 206629:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8287.966659] Lustre: Failing over lustre-OST0003 [ 8288.428891] Lustre: server umount lustre-OST0003 complete [ 8290.467749] LustreError: 203666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8290.503206] LustreError: 203666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8290.793783] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8290.795344] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.824516] Lustre: Skipped 2 previous similar messages [ 8290.964949] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8299.712257] Lustre: DEBUG MARKER: START OST3 again [ 8307.035655] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8307.302911] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8307.310803] Lustre: Skipped 1 previous similar message [ 8307.330575] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8308.635989] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8308.879148] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8308.880564] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8313.652573] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8335.885835] Lustre: Failing over lustre-OST0003 [ 8335.967749] Lustre: server umount lustre-OST0003 complete [ 8337.914176] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.921415] LustreError: 204617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.937392] Lustre: Skipped 1 previous similar message [ 8337.985574] LustreError: 204617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8353.251650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.271085] Lustre: Skipped 4 previous similar messages [ 8363.497023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8363.508588] Lustre: Skipped 7 previous similar messages [ 8367.072143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8367.226345] Lustre: server umount lustre-MDT0000 complete [ 8370.256669] LustreError: 209035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808818 with bad export cookie 11039062416413885820 [ 8370.266737] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.735573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8373.744431] Lustre: Skipped 5 previous similar messages [ 8376.693663] Lustre: server umount lustre-MDT0001 complete [ 8380.191966] Lustre: server umount lustre-OST0000 complete [ 8383.737838] Lustre: server umount lustre-OST0001 complete [ 8390.553803] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 8399.441142] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8436.847331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.078748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8437.128220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8437.207465] Lustre: lustre-MDT0000: new disk, initializing [ 8437.298218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.313057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8442.937701] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8457.248895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8457.400807] Lustre: 211951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8457.455662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8457.459439] Lustre: Skipped 1 previous similar message [ 8457.516922] Lustre: lustre-MDT0001: new disk, initializing [ 8457.712083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8457.730530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8462.629486] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8473.551233] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8480.030851] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8487.410238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.628580] Lustre: 213918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8489.446734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8489.531400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8492.961610] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8501.298509] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8505.902759] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8506.063778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8510.302251] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8510.522629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8515.041910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8515.050162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.051269] LustreError: Skipped 1 previous similar message [ 8515.060643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8515.060652] Lustre: Skipped 4 previous similar messages [ 8515.126704] Lustre: Skipped 1 previous similar message [ 8518.513302] Lustre: server umount lustre-OST0000 complete [ 8524.257693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8536.032945] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8536.216790] Lustre: server umount lustre-MDT0000 complete [ 8540.643100] LustreError: 212796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8540.705621] LustreError: 212796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8541.488979] LustreError: 211942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808989 with bad export cookie 11039062416413888116 [ 8541.501586] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8541.507781] LustreError: 211942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8542.315589] Lustre: server umount lustre-MDT0001 complete [ 8547.537896] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:49:52 (1786808992) [ 8568.694093] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 8582.393222] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8620.104231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8620.257689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8620.281508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8620.286424] Lustre: Skipped 1 previous similar message [ 8620.366690] Lustre: lustre-MDT0000: new disk, initializing [ 8620.369553] Lustre: Skipped 1 previous similar message [ 8620.467698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8620.473848] Lustre: Skipped 1 previous similar message [ 8624.300726] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8634.736717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.805693] Lustre: 218533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8634.984225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8638.967787] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8648.801981] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8655.818466] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8664.264506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8664.794575] Lustre: 220501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8666.095080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8672.237034] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8682.199908] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8686.656261] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8686.826987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8691.181490] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8691.366709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8696.803645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8696.805572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8696.810830] LustreError: Skipped 1 previous similar message [ 8696.815411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8696.829860] Lustre: Skipped 3 previous similar messages [ 8696.872303] Lustre: Skipped 9 previous similar messages [ 8699.620253] Lustre: server umount lustre-OST0000 complete [ 8717.280692] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8717.501203] Lustre: server umount lustre-MDT0000 complete [ 8721.135660] LustreError: 218524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809168 with bad export cookie 11039062416413888998 [ 8721.139528] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8721.144602] LustreError: 218524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8721.574476] Lustre: server umount lustre-MDT0001 complete [ 8734.807350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.989718] LustreError: 222718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.293552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.315138] Lustre: Skipped 5 previous similar messages [ 8742.435135] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8754.165877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.842433] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8767.298532] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8772.672607] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8779.331179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8785.330490] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8792.791843] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8804.331136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8804.335888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8804.336157] Lustre: Skipped 7 previous similar messages [ 8804.353583] LustreError: Skipped 1 previous similar message [ 8805.386917] Lustre: server umount lustre-OST0000 complete [ 8823.264210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8823.436503] Lustre: server umount lustre-MDT0000 complete [ 8826.946103] LustreError: 225394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809274 with bad export cookie 11039062416413889880 [ 8826.952191] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8826.959788] LustreError: 225394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8827.209122] Lustre: server umount lustre-MDT0001 complete [ 8937.506506] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8941.192410] Key type lgssc unregistered [ 8941.594611] LNet: 226383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8941.607933] LNetError: 226383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8941.621936] LNet: Removed LNI 192.168.204.152@tcp [ 8942.513154] Key type .llcrypt unregistered [ 8942.515585] Key type ._llcrypt unregistered [ 8966.585630] Key type ._llcrypt registered [ 8966.587611] Key type .llcrypt registered [ 8966.679248] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 8981.427447] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8982.228096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8982.335129] alg: No test for adler32 (adler32-zlib) [ 8983.669317] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 8984.067168] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 8985.760226] Key type lgssc registered [ 8987.126930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9024.581044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9024.636876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9025.873855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9025.934293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9025.988963] Lustre: lustre-MDT0000: new disk, initializing [ 9026.066282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9026.091145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9032.607917] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9044.153177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9044.351896] Lustre: 229736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9044.454853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9044.459692] Lustre: Skipped 1 previous similar message [ 9044.561243] Lustre: lustre-MDT0001: new disk, initializing [ 9044.630038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9044.680333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9044.700069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9049.543890] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9058.931451] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9064.035054] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9070.539827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9070.819447] Lustre: lustre-OST0000: new disk, initializing [ 9070.824249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9070.830943] Lustre: 231705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9070.911612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9077.900066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9077.914482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9078.121336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9078.141379] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9088.502607] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9093.699500] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9093.932989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9099.865824] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9100.209067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9103.841596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9103.848904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.856544] LustreError: Skipped 1 previous similar message [ 9103.881847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9103.884562] Lustre: Skipped 1 previous similar message [ 9103.905609] Lustre: Skipped 1 previous similar message [ 9109.108209] Lustre: server umount lustre-OST0000 complete [ 9114.083887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9114.098650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9114.109563] Lustre: Skipped 1 previous similar message [ 9119.203955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9119.213026] Lustre: Skipped 1 previous similar message [ 9119.899955] Lustre: server umount lustre-MDT0000 complete [ 9123.547074] LustreError: 229728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809571 with bad export cookie 5513752363337328161 [ 9123.550971] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9123.564664] LustreError: 229728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9123.883247] Lustre: server umount lustre-MDT0001 complete [ 9142.997918] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9152.883408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.251179] LustreError: 234640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9153.279387] LustreError: 234640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9153.329299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.243231] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9158.625549] LustreError: 234641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9163.754461] LustreError: 234640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.491697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.830929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.167331] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9174.405787] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9181.100794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9181.523868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.066322] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9195.769111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9195.955564] Lustre: lustre-OST0001: new disk, initializing [ 9195.963791] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9195.970153] Lustre: 236948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9196.838131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9196.861147] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9196.958543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9201.539256] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9213.731923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9220.442462] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9233.846168] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:01:18 (1786809678) [ 9238.501744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9238.514031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.526633] Lustre: Skipped 1 previous similar message [ 9238.531977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.714324] Lustre: server umount lustre-MDT0000 complete [ 9245.155153] LustreError: 234635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9245.198199] LustreError: 234635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9246.424988] LustreError: 234620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809694 with bad export cookie 5513752363337329043 [ 9246.440530] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9246.452268] LustreError: 234620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9246.776468] Lustre: server umount lustre-MDT0001 complete [ 9261.661607] Lustre: server umount lustre-OST0000 complete [ 9266.656498] Lustre: 227962:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786809698/real 1786809698] req@ffffa0617c47b100 x1873605486033792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786809714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9266.688088] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9266.714437] Lustre: Skipped 3 previous similar messages [ 9267.752269] Lustre: 227962:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786809699/real 1786809699] req@ffffa06145635f80 x1873605486034048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786809715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9268.041579] Lustre: server umount lustre-OST0001 complete [ 9278.575974] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 9290.225372] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9330.428723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9330.945637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9331.003337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9331.170796] Lustre: lustre-MDT0000: new disk, initializing [ 9331.275953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.288205] Lustre: Skipped 1 previous similar message [ 9331.333042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9337.753684] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9350.807113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.057569] Lustre: 240906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9351.158670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9351.168468] Lustre: Skipped 1 previous similar message [ 9351.299985] Lustre: lustre-MDT0001: new disk, initializing [ 9351.540409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9351.553804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9360.157466] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9370.534399] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9377.047972] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9385.245305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.671963] Lustre: lustre-OST0000: new disk, initializing [ 9385.682026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9385.692135] Lustre: 242874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9385.859890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9385.870993] Lustre: Skipped 1 previous similar message [ 9387.613394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9387.625641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9387.810691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9395.633951] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9407.600988] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9414.184884] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9414.388170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9420.464726] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9420.695201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9423.843750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.847195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9423.877440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.898639] Lustre: Skipped 5 previous similar messages [ 9429.398194] Lustre: server umount lustre-OST0000 complete [ 9434.086252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9434.094251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9434.100935] Lustre: Skipped 2 previous similar messages [ 9434.117338] Lustre: Skipped 2 previous similar messages [ 9440.651468] Lustre: server umount lustre-MDT0000 complete [ 9444.323487] LustreError: 240912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.359654] LustreError: 240912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9445.246110] LustreError: 240897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809893 with bad export cookie 5513752363337331178 [ 9445.255549] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.262232] LustreError: 240897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9445.852788] Lustre: server umount lustre-MDT0001 complete [ 9456.751408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.342258] LustreError: 245094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9457.484487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.949417] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9473.317065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9478.541534] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9487.090134] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9492.204475] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9499.012675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9506.150794] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9513.420271] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9661.411931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9661.428237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9661.442497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9661.448621] Lustre: Skipped 2 previous similar messages [ 9672.162570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9672.328850] Lustre: server umount lustre-OST0000 complete [ 9682.924612] LustreError: 247598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9682.948844] LustreError: 247598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9683.133062] Lustre: server umount lustre-MDT0000 complete [ 9687.907881] LustreError: 245076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810135 with bad export cookie 5513752363337332060 [ 9687.916573] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9687.932234] LustreError: 245076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9688.578874] Lustre: server umount lustre-MDT0001 complete [ 9782.084131] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 9786.235408] Key type lgssc unregistered [ 9786.720457] LNet: 248703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9786.732202] LNetError: 248703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9786.753645] LNet: Removed LNI 192.168.204.152@tcp [ 9787.672185] Key type .llcrypt unregistered [ 9787.674095] Key type ._llcrypt unregistered [ 9816.332276] Key type ._llcrypt registered [ 9816.334046] Key type .llcrypt registered [ 9816.432754] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 9834.400406] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9836.039921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9836.149632] alg: No test for adler32 (adler32-zlib) [ 9837.230288] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 9837.445702] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 9839.120221] Key type lgssc registered [ 9841.041470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9893.953677] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9905.902987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9905.954886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9907.207369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9907.235405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9907.331894] Lustre: lustre-MDT0000: new disk, initializing [ 9907.419571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.437477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9912.200226] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9924.479060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.578454] Lustre: 252789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9924.620175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9924.623468] Lustre: Skipped 1 previous similar message [ 9924.683894] Lustre: lustre-MDT0001: new disk, initializing [ 9924.771047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9924.797062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9924.810321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9930.677581] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9935.927737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9944.617683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9944.866904] Lustre: lustre-OST0000: new disk, initializing [ 9944.870622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9944.876205] Lustre: 254420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9944.955899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9947.316855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9947.329468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9947.510383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9951.245526] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9964.242474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9964.391197] Lustre: lustre-OST0001: new disk, initializing [ 9964.397141] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9964.406788] Lustre: 255290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9964.513020] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9970.534708] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9973.433276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9973.453804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9973.579603] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9984.328820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9993.157481] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10009.583801] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10009.587759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10009.614919] Lustre: Skipped 3 previous similar messages [10009.627456] Lustre: Skipped 3 previous similar messages [10014.007247] Lustre: server umount lustre-MDT0000 complete [10018.549547] LustreError: 252780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810466 with bad export cookie 17859470149961734137 [10018.552133] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10018.564207] LustreError: 252780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10019.448762] Lustre: server umount lustre-MDT0001 complete [10035.168256] Lustre: 250295:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786810467/real 1786810467] req@ffffa061582aca80 x1873606381484928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786810483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10035.210959] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.489877] Lustre: server umount lustre-OST0000 complete [10039.968267] Lustre: 250297:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786810471/real 1786810471] req@ffffa061582adf80 x1873606381485184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786810487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10040.929734] Lustre: server umount lustre-OST0001 complete [10055.870989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10067.118124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10077.100802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10088.183319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10112.402833] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10125.519814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.539105] Lustre: lustre-MDT0000: reset Object Index mappings [10126.073396] LustreError: 260305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.090920] LustreError: 260305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10126.172556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10130.239352] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10131.428739] LustreError: 260306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.533879] LustreError: 260305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10139.302025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.316994] Lustre: lustre-MDT0001: reset Object Index mappings [10139.644274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10144.496682] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10148.074802] Lustre: 261450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10156.429252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10156.768944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10164.990125] LustreError: 261804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10166.032358] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10170.159387] LustreError: 262122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.195493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10170.200529] LustreError: 262122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10178.345266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10183.711431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10186.948188] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10195.713882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10199.565472] Lustre: 263325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10205.784674] Lustre: server umount lustre-MDT0000 complete [10206.177472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10206.181841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.193801] LustreError: 260306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10206.208609] LustreError: 260306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10209.262182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10209.291392] Lustre: Skipped 2 previous similar messages [10209.646472] LustreError: 260285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810657 with bad export cookie 17859470149961740927 [10209.656514] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10209.657306] LustreError: 260285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10209.899699] Lustre: server umount lustre-MDT0001 complete [10223.670275] Lustre: server umount lustre-OST0000 complete [10238.019853] Lustre: server umount lustre-OST0001 complete [10260.054157] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:18:25 (1786810705) [10275.933392] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [10287.149495] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10324.701344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.916838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10324.943590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10325.013971] Lustre: lustre-MDT0000: new disk, initializing [10325.079851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10325.088316] Lustre: Skipped 1 previous similar message [10325.099619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10329.255959] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10341.132250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.303882] Lustre: 267578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10341.383635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10341.390164] Lustre: Skipped 1 previous similar message [10341.601859] Lustre: lustre-MDT0001: new disk, initializing [10341.782772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10341.798725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10347.191641] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10357.291765] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10365.145168] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10374.350625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10374.719134] Lustre: lustre-OST0000: new disk, initializing [10374.722543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10374.730656] Lustre: 269552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10375.896864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10375.910713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10376.045108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10380.701376] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10390.447854] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10395.371900] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10395.568723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10399.623752] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10399.882765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10406.888924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10406.893191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.899020] LustreError: Skipped 1 previous similar message [10406.900531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10406.915950] Lustre: Skipped 1 previous similar message [10406.932379] Lustre: Skipped 1 previous similar message [10408.168487] Lustre: server umount lustre-OST0000 complete [10412.011235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10412.026587] Lustre: Skipped 1 previous similar message [10417.124445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10417.132120] Lustre: Skipped 1 previous similar message [10417.862281] Lustre: server umount lustre-MDT0000 complete [10421.220100] LustreError: 267571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810869 with bad export cookie 17859470149961743020 [10421.226396] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10421.229043] LustreError: 267571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10421.562293] Lustre: server umount lustre-MDT0001 complete [10442.495315] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10453.034438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.699267] LustreError: 272486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.717769] LustreError: 272486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10453.740401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10453.750029] Lustre: Skipped 2 previous similar messages [10458.120172] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10465.922876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10470.416144] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10473.622715] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10480.414042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10487.369307] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10495.992858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10496.190683] Lustre: lustre-OST0001: new disk, initializing [10496.198825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10496.209034] Lustre: 274795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10497.482481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10497.488583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10497.612866] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10502.506941] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10513.665836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10518.396393] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10523.923901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10524.065508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10532.838255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10532.853824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10532.865072] Lustre: Skipped 2 previous similar messages [10532.879245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10538.032641] Lustre: server umount lustre-MDT0000 complete [10539.496680] LustreError: 273747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.529024] LustreError: 273747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10542.536867] LustreError: 272468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810990 with bad export cookie 17859470149961743902 [10542.548747] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10542.552785] LustreError: 272468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10542.977398] Lustre: server umount lustre-MDT0001 complete [10557.048530] Lustre: server umount lustre-OST0000 complete [10571.085895] Lustre: server umount lustre-OST0001 complete [10588.295880] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10598.619289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.067561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.074766] Lustre: Skipped 3 previous similar messages [10603.138546] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10604.515924] LustreError: 277866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.528895] LustreError: 277866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10611.995034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.429464] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10619.480791] Lustre: 279009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10627.118817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10634.409948] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10644.432246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10644.524868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10650.147036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10651.421917] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10659.526616] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10663.613610] Lustre: 280886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10669.877507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10670.073725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10681.259327] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:25:26 (1786811126) [10699.234172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10699.243802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10699.251259] Lustre: Skipped 3 previous similar messages [10699.260124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10699.266550] Lustre: Skipped 4 previous similar messages [10702.681939] Lustre: server umount lustre-MDT0000 complete [10706.872711] LustreError: 277847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811154 with bad export cookie 17859470149961750454 [10706.875413] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10706.886151] LustreError: 277847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10713.378766] Lustre: server umount lustre-MDT0001 complete [10722.799594] Lustre: server umount lustre-OST0000 complete [10726.364234] Lustre: server umount lustre-OST0001 complete [10732.387163] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [10742.408229] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10788.620966] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10799.032348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10799.272118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10799.296255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10799.368567] Lustre: lustre-MDT0000: new disk, initializing [10799.509645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10805.554809] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10816.043858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.154836] Lustre: 285127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10816.462411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10822.567973] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10835.970963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.324882] Lustre: 286759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10837.895815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10843.575774] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10856.156026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10856.599765] Lustre: 287630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10856.795783] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10856.808396] Lustre: Skipped 6 previous similar messages [10858.047689] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10858.075883] Lustre: Skipped 1 previous similar message [10858.305933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10863.478568] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10874.226775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10877.833266] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10877.837385] Lustre: Skipped 1 previous similar message [10893.364196] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:28:58 (1786811338) [10898.398739] Lustre: server umount lustre-MDT0000 complete [10898.414716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10898.419691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.432251] Lustre: Skipped 5 previous similar messages [10898.439813] LustreError: 285138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.474560] LustreError: 285138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10901.861838] LustreError: 285119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811349 with bad export cookie 17859470149961758105 [10901.863233] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10901.872763] LustreError: 285119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10902.347808] Lustre: server umount lustre-MDT0001 complete [10917.231827] Lustre: server umount lustre-OST0000 complete [10920.737398] Lustre: 250296:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786811352/real 1786811352] req@ffffa061445bb480 x1873606381801472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786811368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10920.759836] Lustre: 250296:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10921.978584] Lustre: server umount lustre-OST0001 complete [10934.044368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.315291] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10950.078387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.008316] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10963.809066] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10969.802471] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10976.687522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10983.777885] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10990.845235] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11011.739786] loop1: detected capacity change from 0 to 204800000 [11011.911842] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11011.918686] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11011.925251] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11018.456671] loop1: detected capacity change from 0 to 204800000 [11026.187216] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.290684] LustreError: 289860:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14366: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11026.300990] LustreError: 289860:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14366 [11026.304602] LustreError: 289860:0:(mgs_handler.c:517:mgs_target_reg()) fs14366-MDT0001: Error checking target index: rc = -13 [11026.312433] LustreError: fs14366-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11026.318283] LustreError: 293595:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11026.323243] LustreError: 293595:0:(tgt_mount.c:2129:server_put_super()) no obd fs14366-MDT0001 [11026.326631] LustreError: 293595:0:(tgt_mount.c:116:server_deregister_mount()) fs14366-MDT0001 not registered [11026.360232] Lustre: server umount fs14366-MDT0001 complete [11026.365490] LustreError: 293595:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11037.472352] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.569964] Lustre: Setting parameter fs14366-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14366-MDT0001 [11037.608249] Lustre: srv-fs14366-MDT0001: No data found on store. Initialize space. [11037.612593] Lustre: Skipped 4 previous similar messages [11037.695554] Lustre: fs14366-MDT0001: new disk, initializing [11037.699931] Lustre: Skipped 3 previous similar messages [11037.808803] Lustre: srv-fs14366-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11037.814107] Lustre: 294062:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14366-MDT0001-mdtlov: can't create new llog: rc = -115 [11037.822804] LustreError: 294062:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14366-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11043.794540] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11050.916640] Lustre: server umount fs14366-MDT0001 complete [11058.153360] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11063.791451] loop1: detected capacity change from 0 to 204800000 [11070.115314] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.916382] Lustre: 295629:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14366-MDT0001-mdtlov: can't create new llog: rc = -115 [11070.924027] LustreError: 295629:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14366-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11075.524623] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11078.641777] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11080.704529] Lustre: server umount fs14366-MDT0001 complete [11093.996897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.015295] Lustre: Skipped 6 previous similar messages [11094.020376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11094.032162] Lustre: Skipped 8 previous similar messages [11106.272150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11106.607216] Lustre: server umount lustre-MDT0000 complete [11116.275055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.403733] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11121.215586] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11122.171542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11128.830800] loop1: detected capacity change from 0 to 204800000 [11134.711222] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.487941] Lustre: 298079:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14366-MDT0001-mdtlov: can't create new llog: rc = -115 [11135.495159] LustreError: 298079:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14366-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11140.188121] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11145.229312] Lustre: server umount fs14366-MDT0001 complete [11155.605047] LustreError: 291705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11155.618456] LustreError: 291705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 112 previous similar messages [11170.787587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11170.804670] LustreError: Skipped 1 previous similar message [11170.816253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11170.822853] Lustre: Skipped 9 previous similar messages [11176.937972] Lustre: server umount lustre-OST0000 complete [11186.916966] Lustre: server umount lustre-MDT0000 complete [11190.736199] LustreError: 289855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811638 with bad export cookie 17859470149961760786 [11190.742203] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11190.748094] LustreError: 289855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11191.095814] Lustre: server umount lustre-MDT0001 complete [11201.606878] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [11204.462544] Key type lgssc unregistered [11204.826643] LNet: 300425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11204.835366] LNetError: 300425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11204.854869] LNet: Removed LNI 192.168.204.152@tcp [11205.902186] Key type .llcrypt unregistered [11205.906348] Key type ._llcrypt unregistered [11228.904609] Key type ._llcrypt registered [11228.907543] Key type .llcrypt registered [11229.049097] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [11230.464802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11230.595084] alg: No test for adler32 (adler32-zlib) [11231.621712] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [11232.027612] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [11233.736207] Key type lgssc registered [11235.466977] Lustre: Echo OBD driver; http://www.lustre.org/ [11244.422037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11244.480593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.116753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11251.396280] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11255.276805] LustreError: 301696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11255.295459] LustreError: 301696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11260.034779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.387356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11264.462673] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11272.644720] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11279.346266] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11286.594383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11286.945074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11291.050444] LustreError: 303527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.117765] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11296.228226] LustreError: 303745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.359825] LustreError: 303528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.116222] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11305.449597] LustreError: 303527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.476455] LustreError: 303527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11315.359067] LustreError: 303745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.384933] LustreError: 303745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11326.257740] loop1: detected capacity change from 0 to 204800000 [11326.518775] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11326.529056] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11326.539851] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11332.274289] loop1: detected capacity change from 0 to 204800000 [11335.832717] LustreError: 303527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11335.845965] LustreError: 303527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11337.630148] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11337.744675] Lustre: fs29765-OST0000: new disk, initializing [11337.750950] Lustre: srv-fs29765-OST0000: No data found on store. Initialize space. [11337.757664] Lustre: 305433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11337.844636] Lustre: fs29765-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11344.349051] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11353.243153] Lustre: server umount fs29765-OST0000 complete [11365.669766] loop1: detected capacity change from 0 to 204800000 [11370.773870] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.056291] Lustre: fs29765-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11371.676451] LustreError: 304199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11371.723531] LustreError: 304199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [11372.007857] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11377.562071] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11385.333083] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11389.116373] loop2: detected capacity change from 0 to 204800000 [11389.396825] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11389.406891] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11389.423446] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11397.585396] loop2: detected capacity change from 0 to 204800000 [11405.407770] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11405.523410] LustreError: fs29765-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11405.536810] LustreError: fs29765-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11405.549986] LustreError: 308476:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11405.556942] LustreError: 308476:0:(tgt_mount.c:2129:server_put_super()) no obd fs29765-OST0001 [11405.562811] LustreError: 308476:0:(tgt_mount.c:116:server_deregister_mount()) fs29765-OST0001 not registered [11405.605904] Lustre: server umount fs29765-OST0001 complete [11405.609157] LustreError: 308476:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11421.465141] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.629258] Lustre: fs29765-OST0001: new disk, initializing [11421.644365] Lustre: srv-fs29765-OST0001: No data found on store. Initialize space. [11421.672705] Lustre: 309028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11421.800788] Lustre: fs29765-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11428.676361] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11436.533534] LustreError: 303745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.555594] LustreError: 303745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [11438.196416] Lustre: server umount fs29765-OST0000 complete [11448.920701] Lustre: server umount fs29765-OST0001 complete [11461.401980] loop1: detected capacity change from 0 to 204800000 [11467.260261] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11467.406954] Lustre: fs29765-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11473.346466] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11480.645423] loop2: detected capacity change from 0 to 204800000 [11486.651075] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11494.164917] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11504.532962] Lustre: server umount fs29765-OST0000 complete [11513.196496] Lustre: server umount fs29765-OST0001 complete [11523.569134] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11525.601278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11525.610102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11525.628893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11526.633301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.648021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11528.681344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11528.691938] Lustre: Skipped 1 previous similar message [11530.531534] Lustre: server umount lustre-OST0000 complete [11536.867348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11536.877134] LustreError: Skipped 1 previous similar message [11536.880522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11536.894023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.921482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11540.289920] Lustre: server umount lustre-MDT0000 complete [11544.193950] LustreError: 302440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811992 with bad export cookie 18335946132278576661 [11544.203316] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11544.211414] LustreError: 302440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11544.653119] Lustre: server umount lustre-MDT0001 complete [11559.108958] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [11563.605345] Key type lgssc unregistered [11564.156726] LNet: 314953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11564.174472] LNetError: 314953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11564.199499] LNet: Removed LNI 192.168.204.152@tcp [11565.530149] Key type .llcrypt unregistered [11565.536987] Key type ._llcrypt unregistered [11604.751764] Key type ._llcrypt registered [11604.753851] Key type .llcrypt registered [11604.848122] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [11620.394029] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [11622.307494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11622.489417] alg: No test for adler32 (adler32-zlib) [11623.887477] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [11624.080756] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [11625.744165] Key type lgssc registered [11626.858163] Lustre: Echo OBD driver; http://www.lustre.org/ [11668.051975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11668.108143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.332634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11669.356556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11669.432586] Lustre: lustre-MDT0000: new disk, initializing [11669.523506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11669.580907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11673.658407] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11683.795388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.958369] Lustre: 318815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11684.057764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11684.061890] Lustre: Skipped 1 previous similar message [11684.153244] Lustre: lustre-MDT0001: new disk, initializing [11684.277638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11684.338941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11684.355131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11689.078356] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11698.315788] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11704.127580] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11712.140332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.465120] Lustre: lustre-OST0000: new disk, initializing [11712.474436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11712.495974] Lustre: 320785:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11712.629593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11719.845903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11719.872315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11720.128150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11721.166296] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11732.030099] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11734.203239] Lustre: Mounted lustre-client [11735.204257] Lustre: Unmounted lustre-client [11749.859168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.873796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11749.885297] Lustre: Skipped 1 previous similar message [11753.657271] Lustre: server umount lustre-OST0000 complete [11760.099225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.113423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11760.119955] Lustre: Skipped 2 previous similar messages [11764.491557] Lustre: server umount lustre-MDT0000 complete [11768.539079] LustreError: 318808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812216 with bad export cookie 4892938799819717107 [11768.541147] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11768.549507] LustreError: 318808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11768.814834] Lustre: server umount lustre-MDT0001 complete [11781.273198] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [11784.279822] Key type lgssc unregistered [11793.120231] LNet: 1 local NIs in recovery (showing 1): 192.168.204.152@tcp [11793.826971] LNet: 322948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11793.867393] LNetError: 322948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11863.527898] LNet: Removed LNI 192.168.204.152@tcp [11864.411232] Key type .llcrypt unregistered [11864.415363] Key type ._llcrypt unregistered [11886.415193] Key type ._llcrypt registered [11886.416858] Key type .llcrypt registered [11886.607972] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [11909.328729] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [11910.197035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11910.284531] alg: No test for adler32 (adler32-zlib) [11911.410476] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [11911.630786] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [11913.280169] Key type lgssc registered [11914.399083] Lustre: Echo OBD driver; http://www.lustre.org/ [11944.359464] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:46:29 (1786812389) [11950.359967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11950.394468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11951.638918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11951.656399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11951.745805] Lustre: lustre-MDT0000: new disk, initializing [11951.883341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11951.904284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11956.220853] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11967.890775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11968.072948] Lustre: 326172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11968.124781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11968.127688] Lustre: Skipped 1 previous similar message [11968.179867] Lustre: lustre-MDT0001: new disk, initializing [11968.253920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11968.285935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11968.297435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11973.124537] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11982.486416] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11988.154907] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11994.536752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11994.916474] Lustre: lustre-OST0000: new disk, initializing [11994.926563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11994.947815] Lustre: 328140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11995.063869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12001.155468] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12004.924136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12004.931566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12005.051098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12011.369946] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12030.950967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12030.961217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12030.972621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12030.987806] Lustre: Skipped 1 previous similar message [12033.666344] Lustre: server umount lustre-OST0000 complete [12040.161585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12040.165940] LustreError: Skipped 1 previous similar message [12040.169153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12040.180450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12040.195422] Lustre: Skipped 1 previous similar message [12041.192414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.196353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.086782] Lustre: server umount lustre-MDT0000 complete [12048.329505] LustreError: 326165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812496 with bad export cookie 2331089286510625451 [12048.336088] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12048.349028] LustreError: 326165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12048.981820] Lustre: server umount lustre-MDT0001 complete [12060.031926] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12062.562177] Key type lgssc unregistered [12062.848804] LNet: 330049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12062.853914] LNetError: 330049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12062.862891] LNet: Removed LNI 192.168.204.152@tcp [12063.678792] Key type .llcrypt unregistered [12063.683224] Key type ._llcrypt unregistered [12087.817121] Key type ._llcrypt registered [12087.820638] Key type .llcrypt registered [12087.957922] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [12088.858308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12089.048076] alg: No test for adler32 (adler32-zlib) [12090.240511] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [12090.588410] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [12092.296159] Key type lgssc registered [12093.717748] Lustre: Echo OBD driver; http://www.lustre.org/ [12101.529120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12101.542878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12102.907402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12107.331399] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12112.360521] LustreError: 331326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.388811] LustreError: 331326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12116.538349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.175856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12121.145546] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12129.903620] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12136.898261] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12143.698945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12143.966928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12150.942346] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12158.669140] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12171.165769] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12173.159305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12176.868388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.873757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12176.880205] Lustre: Skipped 2 previous similar messages [12176.888395] Lustre: Skipped 2 previous similar messages [12180.642415] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [12181.991349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12181.995474] Lustre: Skipped 1 previous similar message [12185.739673] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [12190.861135] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [12190.874871] Lustre: Skipped 3 previous similar messages [12191.200387] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12191.475299] Lustre: server umount lustre-MDT0000 complete [12192.235644] LustreError: 331326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.255021] LustreError: 331326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12195.234343] LustreError: 331306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812643 with bad export cookie 13263022574758683301 [12195.239663] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12195.243032] LustreError: 331306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12195.984169] LustreError: 331322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.356895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.420181] LustreError: 333165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.420620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12200.437490] LustreError: 333165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12200.457436] Lustre: Skipped 2 previous similar messages [12201.980744] Lustre: server umount lustre-MDT0001 complete [12206.871531] Lustre: server umount lustre-OST0000 complete [12216.693166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.340726] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12225.620041] Lustre: 335509:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.152@tcp [12227.908415] Lustre: server umount MGS complete [12237.674661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.305394] LustreError: 336081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12238.320193] LustreError: 336081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12238.379871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12242.897379] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12248.561038] LustreError: 336081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.581127] LustreError: 336081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12253.129507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.516793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12258.826625] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12269.213092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12269.649586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12275.888465] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12286.181689] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12288.030868] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12295.290303] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12297.162356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12304.871474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12304.888635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12304.902655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12304.910031] Lustre: Skipped 1 previous similar message [12309.469712] Lustre: server umount lustre-OST0000 complete [12315.105417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.108154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12315.127014] Lustre: Skipped 1 previous similar message [12315.138448] LustreError: Skipped 1 previous similar message [12320.227369] LustreError: 336078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12320.250788] LustreError: 336078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12322.327531] Lustre: server umount lustre-MDT0000 complete [12326.146285] LustreError: 336061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812774 with bad export cookie 13263022574758684141 [12326.154789] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12326.157027] LustreError: 336061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12326.420615] Lustre: server umount lustre-MDT0001 complete [12339.376884] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12342.560275] Key type lgssc unregistered [12342.836774] LNet: 339521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12342.845316] LNetError: 339521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12342.870358] LNet: Removed LNI 192.168.204.152@tcp [12343.941103] Key type .llcrypt unregistered [12343.943211] Key type ._llcrypt unregistered [12367.089435] Key type ._llcrypt registered [12367.092768] Key type .llcrypt registered [12367.236424] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [12386.296487] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [12387.353484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12387.432487] alg: No test for adler32 (adler32-zlib) [12388.499434] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [12388.701027] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [12390.392427] Key type lgssc registered [12391.536979] Lustre: Echo OBD driver; http://www.lustre.org/ [12427.807402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12427.887482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.168913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12429.197530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12429.315318] Lustre: lustre-MDT0000: new disk, initializing [12429.422951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12429.444229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12433.569441] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12447.010211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.148096] Lustre: 342683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12447.194544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12447.198384] Lustre: Skipped 1 previous similar message [12447.252814] Lustre: lustre-MDT0001: new disk, initializing [12447.334063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12447.367283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12447.381973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12451.700343] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12462.139091] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12468.700466] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12476.823434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12477.136514] Lustre: lustre-OST0000: new disk, initializing [12477.140610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12477.151791] Lustre: 344650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12477.301153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12483.160270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12483.181329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12483.369155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12484.520983] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12494.226331] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12498.586411] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12498.838473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12504.179851] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12504.345854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12509.155091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12509.156048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.169120] LustreError: Skipped 1 previous similar message [12509.172344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12509.210091] Lustre: Skipped 1 previous similar message [12512.618765] Lustre: server umount lustre-OST0000 complete [12518.883690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12518.892542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12518.921661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12518.930954] Lustre: Skipped 1 previous similar message [12522.460426] Lustre: server umount lustre-MDT0000 complete [12526.014055] LustreError: 342675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812973 with bad export cookie 18074128011690344126 [12526.018469] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12526.025781] LustreError: 342675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12526.329729] Lustre: server umount lustre-MDT0001 complete [12536.724384] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:56:21 (1786812981) [12555.547687] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [12567.141840] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [12610.046653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12610.430546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12610.472830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12610.585518] Lustre: lustre-MDT0000: new disk, initializing [12610.740728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12610.780618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12615.891686] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12630.609772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12630.795329] Lustre: 349411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12630.863340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12630.874273] Lustre: Skipped 1 previous similar message [12630.972366] Lustre: lustre-MDT0001: new disk, initializing [12631.139899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12631.195755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12631.216648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12636.532489] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12646.331598] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12652.957291] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12660.671547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12660.946503] Lustre: lustre-OST0000: new disk, initializing [12660.949562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12660.963256] Lustre: 351382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12661.091846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12667.069864] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12667.469699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12667.477810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12667.593690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12677.729311] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12713.952831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12713.955124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12713.968576] LustreError: Skipped 1 previous similar message [12713.969725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12713.993201] Lustre: Skipped 2 previous similar messages [12713.995179] Lustre: Skipped 2 previous similar messages [12719.076991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12719.082493] Lustre: Skipped 2 previous similar messages [12719.253161] Lustre: server umount lustre-OST0000 complete [12723.765522] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12723.771709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12723.784701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.135462] Lustre: server umount lustre-MDT0000 complete [12734.384810] LustreError: 349403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813182 with bad export cookie 18074128011690345008 [12734.395829] LustreError: 349403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12734.406517] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12734.887036] Lustre: server umount lustre-MDT0001 complete [12746.197122] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12750.236822] Key type lgssc unregistered [12750.625276] LNet: 353557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12750.630952] LNetError: 353557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12750.654765] LNet: Removed LNI 192.168.204.152@tcp [12751.736375] Key type .llcrypt unregistered [12751.743574] Key type ._llcrypt unregistered [12850.082912] Key type ._llcrypt registered [12850.088252] Key type .llcrypt registered [12850.166716] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12850.587445] Key type .llcrypt unregistered [12850.589373] Key type ._llcrypt unregistered [12931.054287] Key type ._llcrypt registered [12931.055649] Key type .llcrypt registered [12931.181771] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12931.506425] Key type .llcrypt unregistered [12931.509676] Key type ._llcrypt unregistered [12968.702818] Key type ._llcrypt registered [12968.708562] Key type .llcrypt registered [12968.799817] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12969.046400] Key type .llcrypt unregistered [12969.048095] Key type ._llcrypt unregistered [13000.642656] Key type ._llcrypt registered [13000.646335] Key type .llcrypt registered [13000.745920] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13001.084380] Key type .llcrypt unregistered [13001.091738] Key type ._llcrypt unregistered [13090.759744] Key type ._llcrypt registered [13090.762331] Key type .llcrypt registered [13090.892696] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13091.178668] Key type .llcrypt unregistered [13091.181451] Key type ._llcrypt unregistered [13128.881774] Key type ._llcrypt registered [13128.884351] Key type .llcrypt registered [13128.985972] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13129.330422] Key type .llcrypt unregistered [13129.337108] Key type ._llcrypt unregistered [13165.328731] Key type ._llcrypt registered [13165.330785] Key type .llcrypt registered [13165.468958] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13165.939544] Key type .llcrypt unregistered [13165.945180] Key type ._llcrypt unregistered [13203.310212] Key type ._llcrypt registered [13203.312489] Key type .llcrypt registered [13203.429257] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13203.701307] Key type .llcrypt unregistered [13203.703086] Key type ._llcrypt unregistered [13241.044675] Key type ._llcrypt registered [13241.051556] Key type .llcrypt registered [13241.151581] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13241.407332] Key type .llcrypt unregistered [13241.413554] Key type ._llcrypt unregistered [13272.364617] Key type ._llcrypt registered [13272.369434] Key type .llcrypt registered [13272.504408] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13272.758420] Key type .llcrypt unregistered [13272.761153] Key type ._llcrypt unregistered [13332.000951] Key type ._llcrypt registered [13332.003225] Key type .llcrypt registered [13332.099933] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13332.315623] Key type .llcrypt unregistered [13332.325123] Key type ._llcrypt unregistered [13365.826365] Key type ._llcrypt registered [13365.831338] Key type .llcrypt registered [13365.978581] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13366.241255] Key type .llcrypt unregistered [13366.243992] Key type ._llcrypt unregistered [13403.878946] Key type ._llcrypt registered [13403.882941] Key type .llcrypt registered [13404.003410] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13404.444535] Key type .llcrypt unregistered [13404.446875] Key type ._llcrypt unregistered [13462.738262] Key type ._llcrypt registered [13462.740495] Key type .llcrypt registered [13462.824419] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [13479.391412] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13480.720739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13480.838982] alg: No test for adler32 (adler32-zlib) [13481.829315] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [13482.083765] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [13483.792168] Key type lgssc registered [13484.936711] Lustre: Echo OBD driver; http://www.lustre.org/ [13523.610282] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13523.643169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13524.879954] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13524.918528] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13525.004066] Lustre: tst250fs-MDT0000: new disk, initializing [13525.087132] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13525.131458] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13530.596071] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13543.497237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13543.695154] Lustre: 373955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13543.776529] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13543.781404] Lustre: Skipped 1 previous similar message [13543.876861] Lustre: tst250fs-MDT0001: new disk, initializing [13543.985560] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13544.023169] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13544.037220] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13548.311315] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13553.162148] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13560.570227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13560.812766] Lustre: tst250fs-OST0000: new disk, initializing [13560.817069] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13560.822935] Lustre: 375591:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13560.917849] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13567.179189] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13570.609886] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13570.624653] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13570.757855] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13577.382611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13577.600757] Lustre: tst250fs-OST0001: new disk, initializing [13577.607959] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13577.621442] Lustre: 376461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13577.732068] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13579.879559] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13579.891688] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13579.959721] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13583.281935] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13588.890579] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13591.272151] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13591.281710] Lustre: Skipped 2 previous similar messages [13610.466226] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13610.470838] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13610.481536] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13610.982412] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13610.992611] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13611.000647] Lustre: Skipped 1 previous similar message [13611.017957] Lustre: Skipped 2 previous similar messages [13615.853683] Lustre: server umount tst250fs-MDT0000 complete [13619.535218] LustreError: 373948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814067 with bad export cookie 6032368514562686238 [13619.542266] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13619.543049] LustreError: 373948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13619.843350] Lustre: server umount tst250fs-MDT0001 complete [13633.761719] Lustre: server umount tst250fs-OST0000 complete [13646.448700] Lustre: server umount tst250fs-OST0001 complete [13663.207916] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [13672.374408] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13702.539348] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:15:47 (1786814147) [13704.152601] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13720.455402] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [13730.117800] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13758.841815] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13508 sec ========= 13:16:44 (1786814204) [13760.606473] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:16:45 (1786814205) === [13764.639978] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:16:49 (1786814209) === [13807.738289] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13811.177935] Key type lgssc unregistered [13811.534508] LNet: 385589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13811.548195] LNetError: 385589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13812.585124] LNet: Removed LNI 192.168.204.152@tcp [13813.776831] Key type .llcrypt unregistered [13813.778594] Key type ._llcrypt unregistered