[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 513216331 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003269] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008597] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012127] LSM: Security Framework initializing [ 0.013059] Yama: becoming mindful. [ 0.015031] SELinux: Initializing. [ 0.016081] *** VALIDATE selinux *** [ 0.023339] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027998] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031102] *** VALIDATE tmpfs *** [ 0.032000] *** VALIDATE proc *** [ 0.032227] *** VALIDATE cgroup *** [ 0.033000] *** VALIDATE cgroup2 *** [ 0.033253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037000] Speculative Store Bypass: Vulnerable [ 0.038000] debug: unmapping init [mem 0xffffffff9b259000-0xffffffff9b260fff] [ 0.040175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041663] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043011] ... bit width: 48 [ 0.044009] ... generic registers: 4 [ 0.045010] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047011] ... fixed-purpose events: 3 [ 0.048012] ... event mask: 000000070000000f [ 0.049278] rcu: Hierarchical SRCU implementation. [ 0.051424] smp: Bringing up secondary CPUs ... [ 0.052571] x86: Booting SMP configuration: [ 0.053026] .... node #0, CPUs: #1 #2 #3 [ 0.058106] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128023] node 0 deferred pages initialised in 65ms [ 0.131096] devtmpfs: initialized [ 0.132195] x86/mm: Memory block size: 128MB [ 0.134650] gcov: version magic: 0x41383552 [ 0.140356] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142268] pinctrl core: initialized pinctrl subsystem [ 0.143182] [ 0.143614] ************************************************************* [ 0.144000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146012] ** ** [ 0.149013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151012] ** ** [ 0.153014] ** This means that this kernel is built to expose internal ** [ 0.156013] ** IOMMU data structures, which may compromise security on ** [ 0.158011] ** your system. ** [ 0.160011] ** ** [ 0.162012] ** If you see this message and you are not debugging the ** [ 0.164018] ** kernel, report this immediately to your vendor! ** [ 0.165010] ** ** [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ************************************************************* [ 0.171655] NET: Registered protocol family 16 [ 0.172000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175437] cpuidle: using governor menu [ 0.176000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177439] PCI: Using configuration type 1 for base access [ 0.178000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188128] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192129] cryptd: max_cpu_qlen set to 1000 [ 0.196154] ACPI: Added _OSI(Module Device) [ 0.197000] ACPI: Added _OSI(Processor Device) [ 0.199017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206746] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213290] ACPI: Interpreter enabled [ 0.214139] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216011] ACPI: Using IOAPIC for interrupt routing [ 0.217161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232519] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.238018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246503] acpiphp: Slot [2] registered [ 0.248125] acpiphp: Slot [3] registered [ 0.250248] acpiphp: Slot [4] registered [ 0.252104] acpiphp: Slot [5] registered [ 0.254089] acpiphp: Slot [6] registered [ 0.256111] acpiphp: Slot [7] registered [ 0.257096] acpiphp: Slot [8] registered [ 0.259096] acpiphp: Slot [9] registered [ 0.260139] acpiphp: Slot [10] registered [ 0.261085] acpiphp: Slot [11] registered [ 0.262086] acpiphp: Slot [12] registered [ 0.264089] acpiphp: Slot [13] registered [ 0.265086] acpiphp: Slot [14] registered [ 0.267094] acpiphp: Slot [15] registered [ 0.268106] acpiphp: Slot [16] registered [ 0.269086] acpiphp: Slot [17] registered [ 0.271093] acpiphp: Slot [18] registered [ 0.272099] acpiphp: Slot [19] registered [ 0.274102] acpiphp: Slot [20] registered [ 0.275084] acpiphp: Slot [21] registered [ 0.276084] acpiphp: Slot [22] registered [ 0.278111] acpiphp: Slot [23] registered [ 0.279074] acpiphp: Slot [24] registered [ 0.281087] acpiphp: Slot [25] registered [ 0.282091] acpiphp: Slot [26] registered [ 0.284100] acpiphp: Slot [27] registered [ 0.285088] acpiphp: Slot [28] registered [ 0.287086] acpiphp: Slot [29] registered [ 0.288113] acpiphp: Slot [30] registered [ 0.290100] acpiphp: Slot [31] registered [ 0.291061] PCI host bridge to bus 0000:00 [ 0.293019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303022] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.305021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311614] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316862] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326772] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.331792] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.334020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.338016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340421] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343074] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.346043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348623] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.355070] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.369016] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.376021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381060] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410014] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.418243] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447026] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.458414] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.474019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.486017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.496640] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.502014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.507014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.520025] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.527000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.531013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.537018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.547015] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.554000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.559015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.565017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.575013] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.585746] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.589037] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.591337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.594377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.595000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.599103] iommu: Default domain type: Passthrough [ 0.600000] SCSI subsystem initialized [ 0.600056] ACPI: bus type USB registered [ 0.601000] usbcore: registered new interface driver usbfs [ 0.603073] usbcore: registered new interface driver hub [ 0.605070] usbcore: registered new device driver usb [ 0.606256] pps_core: LinuxPPS API ver. 1 registered [ 0.608010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.611071] PTP clock support registered [ 0.614027] EDAC MC: Ver: 3.0.0 [ 0.615141] PCI: Using ACPI for IRQ routing [ 0.616000] NetLabel: Initializing [ 0.616000] NetLabel: domain hash size = 128 [ 0.617010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.619073] NetLabel: unlabeled traffic allowed by default [ 0.621237] vgaarb: loaded [ 0.622258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.624018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.635691] clocksource: Switched to clocksource kvm-clock [ 0.738284] VFS: Disk quotas dquot_6.6.0 [ 0.740068] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.742777] *** VALIDATE ramfs *** [ 0.744186] *** VALIDATE hugetlbfs *** [ 0.745819] pnp: PnP ACPI init [ 0.748513] pnp: PnP ACPI: found 6 devices [ 0.769109] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.772379] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.774453] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.776490] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.778860] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.781173] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.784536] NET: Registered protocol family 2 [ 0.787351] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.792546] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.796327] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.801469] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.806135] TCP: Hash tables configured (established 65536 bind 65536) [ 0.810239] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.813511] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.817148] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.820926] NET: Registered protocol family 1 [ 0.826302] RPC: Registered named UNIX socket transport module. [ 0.828356] RPC: Registered udp transport module. [ 0.830223] RPC: Registered tcp transport module. [ 0.832169] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.834642] NET: Registered protocol family 44 [ 0.836061] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.839367] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.841620] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.844167] PCI: CLS 0 bytes, default 64 [ 0.846089] Unpacking initramfs... [ 2.321342] debug: unmapping init [mem 0xffff99557cc54000-0xffff99557ffbffff] [ 2.325357] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.327312] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.329852] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.839847] Initialise system trusted keyrings [ 2.841523] Key type blacklist registered [ 2.843799] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.853478] zbud: loaded [ 2.856635] *** VALIDATE nfs *** [ 2.858050] *** VALIDATE nfs4 *** [ 2.860463] pstore: using deflate compression [ 2.864635] Platform Keyring initialized [ 3.019702] NET: Registered protocol family 38 [ 3.021381] Key type asymmetric registered [ 3.022765] Asymmetric key parser 'x509' registered [ 3.026394] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.031297] io scheduler mq-deadline registered [ 3.033073] io scheduler kyber registered [ 3.034328] io scheduler bfq registered [ 3.036345] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.039994] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.044175] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.048678] ACPI: Power Button [PWRF] [ 3.168749] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.278175] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.467823] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.559747] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.774439] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.808602] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.843667] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.849717] Non-volatile memory driver v1.3 [ 3.852262] Linux agpgart interface v0.103 [ 3.895153] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.898405] vda: detected capacity change from 0 to 68526080 [ 3.920832] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.924280] vdb: detected capacity change from 0 to 1073741824 [ 3.940683] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.943466] vdc: detected capacity change from 0 to 2621440000 [ 3.960169] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.962698] vdd: detected capacity change from 0 to 2621440000 [ 3.979711] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.981940] vde: detected capacity change from 0 to 4294967296 [ 3.998225] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.002806] vdf: detected capacity change from 0 to 4294967296 [ 4.010093] libphy: Fixed MDIO Bus: probed [ 4.016791] usbcore: registered new interface driver usbserial_generic [ 4.019053] usbserial: USB Serial support registered for generic [ 4.021531] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.027226] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.028831] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.032036] mousedev: PS/2 mouse device common for all mice [ 4.034933] rtc_cmos 00:05: RTC can wake from S4 [ 4.037816] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.039591] rtc_cmos 00:05: registered as rtc0 [ 4.043729] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.045588] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.047289] intel_pstate: CPU model not supported [ 4.054452] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.055375] hid: raw HID events driver (C) Jiri Kosina [ 4.061130] usbcore: registered new interface driver usbhid [ 4.062595] usbhid: USB HID core driver [ 4.064272] drop_monitor: Initializing network drop monitor service [ 4.066743] Initializing XFRM netlink socket [ 4.068809] NET: Registered protocol family 10 [ 4.077044] Segment Routing with IPv6 [ 4.078663] NET: Registered protocol family 17 [ 4.081537] mpls_gso: MPLS GSO support [ 4.088526] RAS: Correctable Errors collector initialized. [ 4.090788] AVX version of gcm_enc/dec engaged. [ 4.092828] AES CTR mode by8 optimization enabled [ 4.206557] sched_clock: Marking stable (4206507304, 0)->(5149217675, -942710371) [ 4.209819] registered taskstats version 1 [ 4.212179] Loading compiled-in X.509 certificates [ 4.214324] zswap: loaded using pool lzo/zbud [ 4.242037] Key type big_key registered [ 4.256180] Key type encrypted registered [ 4.257654] ima: No TPM chip found, activating TPM-bypass! [ 4.259635] ima: Allocated hash algorithm: sha1 [ 4.262184] ima: No architecture policies found [ 4.263727] evm: Initialising EVM extended attributes: [ 4.265446] evm: security.selinux [ 4.266608] evm: security.ima [ 4.267741] evm: security.capability [ 4.269438] evm: HMAC attrs: 0x1 [ 4.271897] rtc_cmos 00:05: setting system clock to 2025-11-17 03:12:30 UTC (1763349150) [ 4.277442] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 4.282454] debug: unmapping init [mem 0xffffffff9af82000-0xffffffff9b258fff] [ 4.292110] Write protecting the kernel read-only data: 28672k [ 4.295395] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 4.298186] debug: unmapping init [mem 0xffffffff99f14000-0xffffffff99ffffff] [ 4.349572] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.358322] systemd[1]: Detected virtualization kvm. [ 4.362417] systemd[1]: Detected architecture x86-64. [ 4.364276] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.420297] systemd[1]: No hostname configured. [ 4.422348] systemd[1]: Set hostname to . [ 4.424455] random: systemd: uninitialized urandom read (16 bytes read) [ 4.427103] systemd[1]: Initializing machine ID from random generator. [ 4.511336] random: ln: uninitialized urandom read (6 bytes read) [ 4.627874] random: systemd: uninitialized urandom read (16 bytes read) [ 4.631026] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.635333] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.639401] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.380627] device-mapper: uevent: version 1.0.3 [ 5.383226] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.262249] virtio_net virtio0 ens2: renamed from eth0 [ 6.271565] scsi host0: ata_piix [ 6.277820] random: fast init done [ 6.281671] scsi host1: ata_piix [ 6.286823] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.289689] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.794422] random: crng init done [ 10.797416] random: 7 urandom warning(s) missed due to ratelimiting [ 11.646576] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.308631] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.351290] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.182256] SELinux: Disabled at runtime. [ 17.281555] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.296093] systemd[1]: Detected virtualization kvm. [ 17.298047] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.975210] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.989510] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.021614] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.025950] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.029560] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.073562] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.115816] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ 19.963868] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.793279] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.738715] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.943151] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.453802] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.503546] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.418920] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.400966] NFS: Registering the id_resolver key type [ 28.402947] Key type id_resolver registered [ 28.404563] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-server login: [ 81.333117] hrtimer: interrupt took 8106964 ns [ 83.445508] libcfs: loading out-of-tree module taints kernel. [ 83.548510] Key type ._llcrypt registered [ 83.554031] Key type .llcrypt registered [ 83.664226] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 100.728559] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 101.963311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 101.973474] alg: No test for adler32 (adler32-zlib) [ 103.442358] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 104.465817] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 106.199686] Key type lgssc registered [ 108.256636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.098859] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.658579] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.426560] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.916330] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.722767] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.351606] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 186.005891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.109528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 186.140192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.450529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.525430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.756875] Lustre: lustre-MDT0000: new disk, initializing [ 187.929874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.957580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.583218] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.231930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 207.366857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 207.520084] Lustre: 6513:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.576219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.588467] Lustre: Skipped 1 previous similar message [ 207.773251] Lustre: lustre-MDT0001: new disk, initializing [ 207.891601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.948238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.977655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.030620] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.295606] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.760237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.882199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 228.200252] Lustre: lustre-OST0000: new disk, initializing [ 228.203189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.262618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.017079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.030123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.123146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.246284] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.111921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 248.173478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 248.253938] Lustre: lustre-OST0001: new disk, initializing [ 248.256843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.305648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 253.971427] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.549822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.560202] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.601295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 265.696676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.056596] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.887162] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing check_logdir /tmp/testlogs/ [ 287.672680] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing yml_node [ 292.688371] Lustre: DEBUG MARKER: Client: 2.16.61.42 [ 294.897913] Lustre: DEBUG MARKER: MDS: 2.16.61.42 [ 297.214192] Lustre: DEBUG MARKER: OSS: 2.16.61.42 [ 298.578338] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:17:24 EST 2025 [ 315.559951] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 317.163574] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.283292] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.284453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.336435] Lustre: Skipped 1 previous similar message [ 338.401232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.412357] Lustre: Skipped 6 previous similar messages [ 338.599255] LustreError: 12220:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 338.712342] Lustre: server umount lustre-MDT0000 complete [ 342.624334] LustreError: 7900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349488 with bad export cookie 7427989678428840653 [ 342.626700] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.650056] LustreError: 7900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.522598] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.526620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.540493] Lustre: Skipped 3 previous similar messages [ 347.557655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.560470] Lustre: Skipped 1 previous similar message [ 348.972990] LustreError: 12423:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 348.978037] LustreError: 12423:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 349.148276] Lustre: server umount lustre-MDT0001 complete [ 353.234228] LustreError: 12625:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 353.237037] LustreError: 12625:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 353.296751] Lustre: server umount lustre-OST0000 complete [ 356.957575] LustreError: 12827:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 357.006608] LustreError: 12827:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 357.168692] Lustre: server umount lustre-OST0001 complete [ 363.220192] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 372.449456] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 381.788489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.413401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.219039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 399.900573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 406.476465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 406.579042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.808988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.834496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.920262] Lustre: lustre-MDT0000: new disk, initializing [ 407.011393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.027879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.981948] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 421.319189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 421.393090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 421.513686] Lustre: 15011:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.595263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.598556] Lustre: Skipped 1 previous similar message [ 421.683907] Lustre: lustre-MDT0001: new disk, initializing [ 421.815375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.853160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.867913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.118816] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 433.337798] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 437.384482] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 443.890992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 443.947860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 444.219757] Lustre: lustre-OST0000: new disk, initializing [ 444.231180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 444.339051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.381088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.389778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.553442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.386864] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 456.567785] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 461.185976] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.409287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 465.207665] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 465.428170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 471.011795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.014437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.040602] LustreError: Skipped 1 previous similar message [ 471.042364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.047844] Lustre: Skipped 1 previous similar message [ 473.272116] LustreError: 17952:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 473.278732] LustreError: 17952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 473.397569] Lustre: server umount lustre-OST0000 complete [ 478.177140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 478.185250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.198930] Lustre: Skipped 1 previous similar message [ 481.248842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.251450] Lustre: Skipped 1 previous similar message [ 482.997831] LustreError: 18154:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 483.008714] LustreError: 18154:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 483.118494] Lustre: server umount lustre-MDT0000 complete [ 486.119924] LustreError: 15003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349632 with bad export cookie 7427989678428842193 [ 486.125754] LustreError: 15003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 486.127991] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.352957] Lustre: server umount lustre-MDT0001 complete [ 491.156584] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:20:36 (1763349636) [ 508.502510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.043596] LustreError: 19490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.061376] LustreError: 19490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 509.129542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.732682] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 514.531199] LustreError: 19490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.648521] LustreError: 19490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.948355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 526.065342] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 532.858019] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 537.632044] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 540.308488] Lustre: Failing over lustre-MDT0000 [ 540.573853] LustreError: 20833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 540.596308] LustreError: 20833:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 540.826324] Lustre: server umount lustre-MDT0000 complete [ 541.672094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 541.687812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.713081] Lustre: Skipped 1 previous similar message [ 541.717141] LustreError: 19491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.785630] LustreError: 20447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.822505] LustreError: 20447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 557.025497] LustreError: 19485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.034319] LustreError: 19485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 559.018284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.148752] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.266473] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 564.706995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 564.707812] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 564.793174] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 569.126793] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 573.928568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.940793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.951661] Lustre: Skipped 2 previous similar messages [ 579.275125] LustreError: 21814:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 579.284048] LustreError: 21814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 579.417495] Lustre: server umount lustre-MDT0000 complete [ 582.586536] LustreError: 19471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349728 with bad export cookie 7427989678428843607 [ 582.592946] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.600779] LustreError: 19471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 583.080886] Lustre: server umount lustre-MDT0001 complete [ 590.318844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.760097] LustreError: 22491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.851687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.856691] Lustre: Skipped 2 previous similar messages [ 594.081341] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 602.014452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 606.088046] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 610.662549] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 613.894793] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 615.855482] Lustre: Failing over lustre-MDT0000 [ 616.089253] Lustre: server umount lustre-MDT0000 complete [ 616.937895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.173445] LustreError: 22491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.200426] LustreError: 22491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 633.056222] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349763/real 1763349763] req@ffff9955fc072a00 x1849005703349248/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 633.092321] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.317042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.552101] LustreError: 3653:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9955fc070700 x1849005703352064/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 644.206489] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 647.143121] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 647.148981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 647.185175] Lustre: Skipped 1 previous similar message [ 647.198355] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 648.258885] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 655.704497] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 659.357929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.360718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 659.376146] Lustre: Skipped 2 previous similar messages [ 659.395861] Lustre: Skipped 4 previous similar messages [ 665.503247] LustreError: 24816:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 665.505429] LustreError: 24816:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 665.595764] Lustre: server umount lustre-MDT0000 complete [ 669.108276] LustreError: 23835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349815 with bad export cookie 7427989678428844538 [ 669.110358] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.149097] LustreError: 23835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 669.485019] Lustre: server umount lustre-MDT0001 complete [ 676.848302] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:23:42 (1763349822) [ 689.794392] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 697.294611] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 706.475591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 713.923416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 719.693487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 725.775814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 733.533644] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 743.074543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 743.115695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.278862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 743.303810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 743.360380] Lustre: lustre-MDT0000: new disk, initializing [ 743.448248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.451865] Lustre: Skipped 2 previous similar messages [ 743.462944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 747.268559] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 756.067752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 756.143026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 756.227817] Lustre: 28490:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 756.472174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 759.632822] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 766.540682] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 769.841115] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 776.398487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 776.460388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 778.705320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 778.769947] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 778.990773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 779.027371] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 782.953489] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 790.028234] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.493371] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 814.563096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.571609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.605229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.608229] Lustre: Skipped 2 previous similar messages [ 816.943064] LustreError: 31145:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 816.951698] LustreError: 31145:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 817.065636] Lustre: server umount lustre-OST0000 complete [ 826.499554] Lustre: server umount lustre-MDT0000 complete [ 830.125577] LustreError: 28480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349976 with bad export cookie 7427989678428844937 [ 830.129925] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 830.130942] LustreError: 28480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 830.594560] Lustre: server umount lustre-MDT0001 complete [ 840.944154] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 843.991861] Key type lgssc unregistered [ 844.290494] LNet: 32083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.300328] LNetError: 32083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.317975] LNet: Removed LNI 192.168.203.154@tcp [ 845.381414] Key type .llcrypt unregistered [ 845.386439] Key type ._llcrypt unregistered [ 867.832501] Key type ._llcrypt registered [ 867.834415] Key type .llcrypt registered [ 867.912395] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 868.830414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 868.897601] alg: No test for adler32 (adler32-zlib) [ 869.933567] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 870.116120] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 871.767173] Key type lgssc registered [ 872.600214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 879.116199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 879.148968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.534425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.991295] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 889.825668] LustreError: 33476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.868475] LustreError: 33476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 891.242815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 891.473926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.532872] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 898.713287] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 901.670623] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.867317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 907.138612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.808167] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 916.101432] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 916.351417] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 938.496724] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:28:04 (1763350084) [ 940.321153] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 941.787885] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 947.364598] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:28:13 (1763350093) [ 954.108450] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:28:19 (1763350099) [ 955.236601] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 962.699666] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 962.703990] Lustre: Skipped 2 previous similar messages [ 964.246728] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:28:29 (1763350109) [ 967.200487] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 967.203107] Lustre: Skipped 3 previous similar messages [ 982.481826] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 982.488385] Lustre: Skipped 6 previous similar messages [ 984.771769] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:28:50 (1763350130) [ 987.343285] Lustre: *** cfs_fail_loc=131b, val=2*** [ 990.445069] Lustre: *** cfs_fail_loc=131b, val=2*** [ 997.353751] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:29:03 (1763350143) [ 998.837190] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 998.840754] Lustre: Skipped 3 previous similar messages [ 1009.738122] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:29:15 (1763350155) [ 1022.948919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.949039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.952604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.961330] Lustre: Skipped 1 previous similar message [ 1025.662129] LustreError: 40071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1025.806931] Lustre: server umount lustre-OST0000 complete [ 1029.600339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1029.612374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.647323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.660500] Lustre: Skipped 1 previous similar message [ 1033.185498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.185720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.209038] Lustre: Skipped 1 previous similar message [ 1035.130375] LustreError: 40273:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1035.137613] LustreError: 40273:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1035.223707] Lustre: server umount lustre-MDT0000 complete [ 1038.208793] LustreError: 33458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350184 with bad export cookie 15141011546983873733 [ 1038.213249] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1038.219568] LustreError: 33458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1038.341275] LustreError: 40475:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1038.349811] LustreError: 40475:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1038.520654] Lustre: server umount lustre-MDT0001 complete [ 1045.968520] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1047.890181] Key type lgssc unregistered [ 1048.152192] LNet: 41006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.156476] LNetError: 41006:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.173607] LNet: Removed LNI 192.168.203.154@tcp [ 1048.741663] Key type .llcrypt unregistered [ 1048.743551] Key type ._llcrypt unregistered [ 1062.970310] Key type ._llcrypt registered [ 1062.971964] Key type .llcrypt registered [ 1063.066917] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 1075.710384] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1076.498588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.513476] alg: No test for adler32 (adler32-zlib) [ 1077.535537] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1077.691470] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1079.319151] Key type lgssc registered [ 1080.213154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1086.910780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1092.026807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1097.156068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1103.070652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1109.329429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1109.379225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.393434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.645658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1110.682959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1110.800545] Lustre: lustre-MDT0000: new disk, initializing [ 1110.880937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.896144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1114.628505] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1123.338080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1123.393347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1123.480375] Lustre: 44134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1123.552595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1123.559644] Lustre: Skipped 1 previous similar message [ 1123.673629] Lustre: lustre-MDT0001: new disk, initializing [ 1123.775438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.819918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1123.842669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1126.994324] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1133.247496] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1135.849541] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1141.029316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1141.070416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1141.247152] Lustre: lustre-OST0000: new disk, initializing [ 1141.250789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1141.318860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.740249] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1147.449338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1147.465188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1147.521762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1152.159924] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1155.304714] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1155.476702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1158.973208] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1159.144539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1163.236634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1163.238410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.240144] LustreError: Skipped 1 previous similar message [ 1163.244813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.247951] Lustre: Skipped 1 previous similar message [ 1166.653400] LustreError: 47075:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1166.759056] Lustre: server umount lustre-OST0000 complete [ 1169.887840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1169.892612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.900671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.907106] Lustre: Skipped 1 previous similar message [ 1173.474955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.481861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.650383] LustreError: 47276:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1175.652996] LustreError: 47276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1175.743814] Lustre: server umount lustre-MDT0000 complete [ 1178.994240] LustreError: 44125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350325 with bad export cookie 7977034157842216234 [ 1178.995709] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.015201] LustreError: 44125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1179.110440] LustreError: 47477:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1179.117383] LustreError: 47477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1179.279871] Lustre: server umount lustre-MDT0001 complete [ 1187.745512] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:32:13 (1763350333) [ 1193.565583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.830532] LustreError: 48197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.850902] LustreError: 48197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1193.897820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.992076] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1199.072617] LustreError: 48197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.193837] LustreError: 48197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.198899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1204.600371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.940873] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1212.853507] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1216.664736] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1223.420595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1223.642100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.633709] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1233.282757] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1247.938712] LustreError: 50412:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1249.030617] Lustre: Modifying parameter general.timeout=1 in log params [ 1250.674291] Lustre: Modifying parameter general.timeout=3 in log params [ 1250.676580] Lustre: Skipped 1 previous similar message [ 1253.553889] Lustre: Modifying parameter general.timeout=6 in log params [ 1253.556724] Lustre: Skipped 2 previous similar messages [ 1258.072155] Lustre: Modifying parameter general.timeout=10 in log params [ 1258.084783] Lustre: Skipped 3 previous similar messages [ 1266.562306] Lustre: Modifying parameter general.timeout=18 in log params [ 1266.564237] Lustre: Skipped 7 previous similar messages [ 1283.333310] Lustre: Modifying parameter general.timeout=34 in log params [ 1283.336251] Lustre: Skipped 15 previous similar messages [ 1316.097195] Lustre: Modifying parameter general.timeout=77 in log params [ 1316.100465] Lustre: Skipped 42 previous similar messages [ 1366.542158] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:35:12 (1763350512) [ 1369.124794] Lustre: 57352:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1370.868718] Lustre: 57450:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1374.051666] Lustre: 57647:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1374.064518] Lustre: 57647:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1380.372287] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1380.382819] Lustre: Skipped 61 previous similar messages [ 1400.034855] Lustre: 59226:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1400.043684] Lustre: 59226:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1400.070304] LustreError: 59226:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1404.648716] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:35:49 (1763350549) [ 1416.160984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.169382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.182722] Lustre: Skipped 2 previous similar messages [ 1416.207790] Lustre: Skipped 3 previous similar messages [ 1421.281394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.290718] Lustre: Skipped 2 previous similar messages [ 1421.414257] LustreError: 59927:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1421.416815] LustreError: 59927:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1421.555288] Lustre: server umount lustre-MDT0000 complete [ 1424.926750] LustreError: 48177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350571 with bad export cookie 7977034157842217116 [ 1424.928852] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.940498] LustreError: 48177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1425.319968] Lustre: server umount lustre-MDT0001 complete [ 1439.284030] LustreError: 60329:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1439.289504] LustreError: 60329:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1439.491328] Lustre: server umount lustre-OST0000 complete [ 1448.117140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1451.908486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.270136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1458.768466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1465.790521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1465.965986] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1466.141610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.592209] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1477.219737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.429223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1477.457201] Lustre: 61781:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1482.253401] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1487.616456] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1491.121275] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1498.154522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1498.486145] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1498.917386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.920698] Lustre: Skipped 1 previous similar message [ 1505.373081] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1511.659908] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1513.153135] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1513.162266] Lustre: Skipped 14 previous similar messages [ 1513.230103] Lustre: 64014:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1522.145966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.146309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.148783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.158895] Lustre: Skipped 1 previous similar message [ 1522.174163] LustreError: Skipped 1 previous similar message [ 1525.575228] LustreError: 64270:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1525.587134] LustreError: 64270:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1525.698225] Lustre: server umount lustre-OST0000 complete [ 1529.324782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1532.385602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.402796] Lustre: Skipped 1 previous similar message [ 1535.747818] Lustre: server umount lustre-MDT0000 complete [ 1537.519973] LustreError: 61794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.532637] LustreError: 61794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1539.432628] LustreError: 61773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350685 with bad export cookie 7977034157842220119 [ 1539.435688] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.453838] LustreError: 61773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1540.242996] Lustre: server umount lustre-MDT0001 complete [ 1552.619702] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1556.172828] Key type lgssc unregistered [ 1556.550664] LNet: 65207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.569028] LNetError: 65207:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.611309] LNet: Removed LNI 192.168.203.154@tcp [ 1557.466856] Key type .llcrypt unregistered [ 1557.471715] Key type ._llcrypt unregistered [ 1582.268605] Key type ._llcrypt registered [ 1582.270861] Key type .llcrypt registered [ 1582.371713] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1583.210477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1583.394517] alg: No test for adler32 (adler32-zlib) [ 1584.455213] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1584.648818] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1586.319249] Key type lgssc registered [ 1587.361617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.217066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.267772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.893050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.940239] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1607.149053] LustreError: 66608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.179731] LustreError: 66608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1610.284325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1611.155661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1615.074748] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1620.273902] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1623.885596] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1630.373995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1630.838966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.790651] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1642.910575] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1650.184427] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1657.828710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.836595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1657.842585] Lustre: Skipped 2 previous similar messages [ 1657.855540] Lustre: Skipped 2 previous similar messages [ 1662.946783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.952537] Lustre: Skipped 2 previous similar messages [ 1663.944065] LustreError: 69318:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1664.009387] Lustre: server umount lustre-MDT0000 complete [ 1667.121581] LustreError: 66588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350813 with bad export cookie 2618870778355791303 [ 1667.137224] LustreError: 66588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1667.140742] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.266387] LustreError: 69520:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1667.277432] LustreError: 69520:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1667.461854] Lustre: server umount lustre-MDT0001 complete [ 1681.243379] LustreError: 69721:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1681.252987] LustreError: 69721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1681.416228] Lustre: server umount lustre-OST0000 complete [ 1689.361651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1692.473649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1695.756450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1699.455029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1706.429980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.782933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1706.830738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1707.186713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.554591] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1719.088299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1719.163380] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1719.213301] Lustre: 71174:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1719.721984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.305489] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1728.274695] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1731.784149] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1737.669327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1737.820105] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1738.161657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.788459] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1747.235271] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1748.593935] Lustre: 73417:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1750.304718] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1755.617888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1755.622506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.633632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.128199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1760.152893] LustreError: 73674:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1760.159763] LustreError: 73674:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1760.365571] Lustre: server umount lustre-OST0000 complete [ 1765.857208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1765.860750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.891827] Lustre: Skipped 1 previous similar message [ 1765.894564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.897309] Lustre: Skipped 2 previous similar messages [ 1771.489035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.496701] Lustre: Skipped 3 previous similar messages [ 1778.143154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1778.226962] LustreError: 73876:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1778.235522] LustreError: 73876:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1778.309820] Lustre: server umount lustre-MDT0000 complete [ 1781.683714] LustreError: 71906:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350927 with bad export cookie 2618870778355792150 [ 1781.688765] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1781.696813] LustreError: 71906:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1782.076241] Lustre: server umount lustre-MDT0001 complete [ 1791.342615] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1793.793902] Key type lgssc unregistered [ 1794.048404] LNet: 74609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.052317] LNetError: 74609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.065325] LNet: Removed LNI 192.168.203.154@tcp [ 1794.733194] Key type .llcrypt unregistered [ 1794.736265] Key type ._llcrypt unregistered [ 1813.859490] Key type ._llcrypt registered [ 1813.861753] Key type .llcrypt registered [ 1813.958702] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1814.593053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.654377] alg: No test for adler32 (adler32-zlib) [ 1815.673488] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1815.898837] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1817.543802] Key type lgssc registered [ 1818.382174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.017654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.035160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.555788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.578652] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1834.979370] LustreError: 75982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.012726] LustreError: 75982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1835.847270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.578857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1839.590259] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1843.842781] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1846.645594] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1852.128653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1852.659679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.809767] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1862.250943] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1877.124841] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1877.888286] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1879.417339] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1879.421394] Lustre: Skipped 1 previous similar message [ 1881.714929] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1881.718817] Lustre: Skipped 2 previous similar messages [ 1885.816149] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1885.826534] Lustre: Skipped 4 previous similar messages [ 1894.392656] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1894.395654] Lustre: Skipped 10 previous similar messages [ 1910.503070] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1910.505892] Lustre: Skipped 21 previous similar messages [ 1942.953703] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1942.957101] Lustre: Skipped 43 previous similar messages [ 1957.540991] Lustre: 83762:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1959.904976] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:45:05 (1763351105) [ 1961.318289] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1962.902577] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:45:08 (1763351108) [ 1967.586974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.592715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1967.596744] Lustre: Skipped 1 previous similar message [ 1967.604291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.707191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.712873] Lustre: Skipped 1 previous similar message [ 1977.832717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.838439] Lustre: Skipped 2 previous similar messages [ 1979.874889] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1979.932235] LustreError: 84057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1980.047630] Lustre: server umount lustre-OST0000 complete [ 1984.992420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1985.000488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.016240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.068974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.194650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.210024] Lustre: Skipped 3 previous similar messages [ 1997.283310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1997.392327] LustreError: 84261:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1997.402783] LustreError: 84261:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1997.504558] Lustre: server umount lustre-MDT0000 complete [ 1998.324446] LustreError: 76696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.333026] LustreError: 76696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2000.693963] LustreError: 75962:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351146 with bad export cookie 15149734006518696277 [ 2000.697385] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.706545] LustreError: 75962:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2000.835745] LustreError: 84463:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2000.843766] LustreError: 84463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2001.043873] Lustre: server umount lustre-MDT0001 complete [ 2010.420165] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2013.277866] Key type lgssc unregistered [ 2013.581986] LNet: 84995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.591809] LNetError: 84995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.611187] LNet: Removed LNI 192.168.203.154@tcp [ 2014.561889] Key type .llcrypt unregistered [ 2014.564741] Key type ._llcrypt unregistered [ 2018.884237] Key type ._llcrypt registered [ 2018.885969] Key type .llcrypt registered [ 2018.978757] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2022.967696] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules [ 2023.938684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.049892] alg: No test for adler32 (adler32-zlib) [ 2025.057976] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2025.273913] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2026.967201] Key type lgssc registered [ 2028.176408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2029.154421] LustreError: 86161:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 2044.424993] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2051.543920] LustreError: 86161:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2053.969395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2053.988062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.598473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.089139] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2064.875776] LustreError: 87169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.892453] LustreError: 87169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2066.941309] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:46:52 (1763351212) [ 2070.002455] LustreError: 87169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.109779] LustreError: 87201:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2071.122188] LustreError: 87201:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2071.134756] LustreError: 87201:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2071.281565] LustreError: 87976:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2071.460133] Lustre: server umount lustre-MDT0000 complete [ 2083.129984] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2085.431856] Key type lgssc unregistered [ 2085.683753] LNet: 88661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.696425] LNetError: 88661:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.712697] LNet: Removed LNI 192.168.203.154@tcp [ 2086.308929] Key type .llcrypt unregistered [ 2086.311301] Key type ._llcrypt unregistered [ 2101.254401] Key type ._llcrypt registered [ 2101.258576] Key type .llcrypt registered [ 2101.346770] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2102.228351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2102.304932] alg: No test for adler32 (adler32-zlib) [ 2103.374256] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2103.585896] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2105.303189] Key type lgssc registered [ 2106.305568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2112.558298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2112.576489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.041949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.416616] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2124.264496] LustreError: 89859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.270761] LustreError: 89859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2126.027510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2126.975789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2130.375337] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2134.984577] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.550149] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2145.559875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2146.228505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.261544] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2157.308316] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2200.934780] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:49:06 (1763351346) [ 2203.160868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2204.917952] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:49:10 (1763351350) [ 2208.736685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2208.754191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.767890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.786772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.786851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.390479] LustreError: 92767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2214.625139] Lustre: server umount lustre-MDT0000 complete [ 2215.914910] LustreError: 89859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.941434] LustreError: 89859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2217.993163] LustreError: 89839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351364 with bad export cookie 15253207040892377423 [ 2217.996251] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.014497] LustreError: 89839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2218.194779] LustreError: 92968:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2218.198668] LustreError: 92968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2218.390631] Lustre: server umount lustre-MDT0001 complete [ 2232.328311] LustreError: 93169:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2232.332417] LustreError: 93169:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2232.650747] Lustre: server umount lustre-OST0000 complete [ 2244.039252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.545495] LustreError: 93799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.743903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.281630] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2250.209334] LustreError: 93799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.310430] LustreError: 93799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.624467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2256.293481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.768568] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2264.143830] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2266.731541] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2271.684860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2277.620145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2277.699372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2277.870274] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2277.875307] LustreError: 93789:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2277.890982] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2277.895756] LustreError: 95663:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2277.902849] LustreError: 95663:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2277.907391] LustreError: 95663:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2277.979670] Lustre: server umount lustre-OST0000 complete [ 2277.981781] LustreError: 95663:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2285.413766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2285.595657] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2285.600582] LustreError: 93788:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2285.616920] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2285.621678] LustreError: 96124:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2285.625286] LustreError: 96124:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2285.628330] LustreError: 96124:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2285.688639] Lustre: server umount lustre-OST0000 complete [ 2285.692473] LustreError: 96124:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2287.727651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2293.109631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2293.252887] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2293.255940] Lustre: Found index 0 for lustre-OST0000, updating log [ 2293.263762] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2293.335041] Lustre: lustre-OST0000: new disk, initializing [ 2293.342384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2293.549748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.313913] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2301.511605] Lustre: Failing over lustre-OST0000 [ 2301.553290] LustreError: 97211:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2301.555904] LustreError: 97211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2301.716234] Lustre: server umount lustre-OST0000 complete [ 2307.044686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.050842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.059147] Lustre: Skipped 1 previous similar message [ 2307.062487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.065668] Lustre: Skipped 2 previous similar messages [ 2311.661373] LustreError: 97412:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2311.666742] LustreError: 97412:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2311.730239] Lustre: server umount lustre-MDT0000 complete [ 2312.161084] LustreError: 93798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.181593] LustreError: 93798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2315.255324] LustreError: 93780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351461 with bad export cookie 15253207040892380167 [ 2315.274111] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.615608] Lustre: server umount lustre-MDT0001 complete [ 2323.605220] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:51:09 (1763351469) [ 2336.768979] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2345.578669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.038249] LustreError: 98980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.177078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.619445] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2357.821524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2362.061948] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2364.305056] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2370.601422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2376.264509] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2384.477949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.587819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2384.642153] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2384.714205] Lustre: lustre-OST0001: new disk, initializing [ 2384.723271] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2385.144080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2385.148520] Lustre: Skipped 2 previous similar messages [ 2390.105329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2390.113086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2390.235139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2390.287269] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2396.730212] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2401.777576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2413.016057] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 2413.019373] Lustre: Skipped 2 previous similar messages [ 2415.078110] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.093040] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.106192] Lustre: Skipped 1 previous similar message [ 2417.587799] LustreError: 102220:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2417.596318] LustreError: 102220:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2417.694833] Lustre: server umount lustre-MDT0001 complete [ 2418.121490] LustreError: 98974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.134078] LustreError: 98974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2419.938945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.658346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2425.818331] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2425.821245] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2425.881691] Lustre: 98968:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2427.763111] LustreError: 102833:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2427.766823] LustreError: 102833:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2427.770968] LustreError: 98971:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2427.881471] LustreError: 102843:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.154@tcp: cfg command failed: rc = -17 [ 2427.901134] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2427.901134] [ 2429.761385] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2431.985317] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2438.321693] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 22:53:03 (1763351583) [ 2452.684897] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2468.259405] Lustre: 104832:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2487.333832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2490.747800] Lustre: 105967:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2574.823506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.839975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.869633] Lustre: Skipped 2 previous similar messages [ 2574.873304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.883843] Lustre: Skipped 4 previous similar messages [ 2577.566742] LustreError: 106425:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2577.575628] LustreError: 106425:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2577.647146] Lustre: server umount lustre-MDT0000 complete [ 2580.363081] LustreError: 98961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351726 with bad export cookie 15253207040892381091 [ 2580.368558] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.372436] LustreError: 98961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2580.451618] LustreError: 98974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.460629] LustreError: 98974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2580.642557] Lustre: server umount lustre-MDT0001 complete [ 2589.201714] Lustre: server umount lustre-OST0000 complete [ 2592.409465] Lustre: server umount lustre-OST0001 complete [ 2596.683479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2603.348490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2604.077207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2610.635936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2617.242655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2617.862511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2632.057168] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2639.433703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.450028] Lustre: lustre-MDT0000: reset Object Index mappings [ 2639.831150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.835447] Lustre: Skipped 1 previous similar message [ 2642.330990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2648.051956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2648.069819] Lustre: lustre-MDT0001: reset Object Index mappings [ 2651.383142] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2653.287361] Lustre: 111017:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2656.988392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2658.408329] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2660.388813] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2664.957101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2666.784300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2668.883986] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2668.905377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2671.457449] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2677.363613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2680.167228] Lustre: 112874:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2720.165220] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:57:45 (1763351865) [ 2749.413899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.422796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.424397] Lustre: Skipped 7 previous similar messages [ 2749.430432] Lustre: Skipped 9 previous similar messages [ 2752.259966] LustreError: 113225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2752.266857] LustreError: 113225:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2752.394663] Lustre: server umount lustre-MDT0000 complete [ 2754.532172] LustreError: 109904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.540993] LustreError: 109904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2755.248481] LustreError: 109885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351901 with bad export cookie 15253207040892427942 [ 2755.251571] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.266518] LustreError: 109885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2761.660188] Lustre: server umount lustre-MDT0001 complete [ 2770.599325] Lustre: server umount lustre-OST0000 complete [ 2787.295455] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2787.411583] Lustre: server umount lustre-OST0001 complete [ 2791.443658] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 2797.478478] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2803.626764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.054746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2812.314928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2816.711713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.786431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.828886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.978238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2821.004105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2821.047512] Lustre: lustre-MDT0000: new disk, initializing [ 2821.120864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.125047] Lustre: Skipped 3 previous similar messages [ 2821.141233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2823.506703] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2828.270881] LustreError: 116581:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2828.275867] LustreError: 116581:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2828.391994] Lustre: server umount lustre-MDT0000 complete [ 2830.290656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2834.498199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2834.577331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.836139] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2834.853248] Lustre: 117101:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2834.868300] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2837.879147] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2841.308959] Lustre: server umount lustre-MDT0000 complete [ 2847.475082] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:59:53 (1763351993) [ 2848.560673] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2850.040562] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:59:55 (1763351995) [ 2861.718341] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2869.267426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.008963] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2877.705045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2877.744622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2877.815308] Lustre: 119118:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2877.865768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2877.868946] Lustre: Skipped 1 previous similar message [ 2877.912945] Lustre: lustre-MDT0001: new disk, initializing [ 2878.012420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2878.017397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2880.672355] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2884.447709] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2889.102923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2889.164204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2889.397879] Lustre: lustre-OST0000: new disk, initializing [ 2889.411971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2890.779961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2890.784797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2890.865209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2892.772451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2900.058575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2900.099925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2900.199837] Lustre: lustre-OST0001: new disk, initializing [ 2900.202865] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2902.172636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2902.179893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2902.261583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2904.509729] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2911.301352] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2913.307897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3181.624238] Lustre: 127916:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3181.630307] Lustre: 127916:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3181.634235] Lustre: 127916:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3181.640783] Lustre: 127916:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3181.646799] Lustre: 127916:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3181.650401] Lustre: 127916:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3354.531603] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:08:19 (1763352499) [ 3368.420740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.424602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.437461] Lustre: Skipped 4 previous similar messages [ 3368.440929] Lustre: Skipped 4 previous similar messages [ 3373.890096] LustreError: 128137:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3373.895544] LustreError: 128137:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3374.017781] Lustre: server umount lustre-MDT0000 complete [ 3376.565774] LustreError: 127929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352522 with bad export cookie 15253207040892491005 [ 3376.574475] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3376.587543] LustreError: 127929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3378.656199] LustreError: 122332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.664683] LustreError: 122332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3382.984668] Lustre: server umount lustre-MDT0001 complete [ 3391.637073] Lustre: server umount lustre-OST0000 complete [ 3408.351286] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3408.507052] Lustre: server umount lustre-OST0001 complete [ 3413.124172] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3420.307665] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3427.189233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3431.810405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.738749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3440.075960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3444.342937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3444.375479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.527311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3444.556989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3444.605525] Lustre: lustre-MDT0000: new disk, initializing [ 3444.668858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.671869] Lustre: Skipped 5 previous similar messages [ 3444.682324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3447.248600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3454.419316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.467456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3454.528592] Lustre: 130928:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3454.745588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3457.385893] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3462.337200] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3464.673405] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3469.232753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3469.280962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3471.333876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3471.344623] Lustre: Skipped 1 previous similar message [ 3471.349132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3471.432990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3473.517724] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3478.931643] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3485.502422] Lustre: lustre-MDD0000: changelog on [ 3487.166771] Lustre: lustre-MDD0001: changelog on [ 4418.764390] Lustre: lustre-MDD0001: changelog off [ 4419.628307] Lustre: 135724:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4420.078714] Lustre: lustre-MDD0000: changelog off [ 4423.137278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4423.145464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.151919] Lustre: Skipped 4 previous similar messages [ 4423.156352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4423.158409] Lustre: Skipped 11 previous similar messages [ 4423.650185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4428.403135] LustreError: 140588:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4428.406217] LustreError: 140588:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4428.474964] Lustre: server umount lustre-OST0000 complete [ 4432.864819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4432.868647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.874386] Lustre: Skipped 1 previous similar message [ 4432.876952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.879851] Lustre: Skipped 1 previous similar message [ 4436.555287] Lustre: server umount lustre-MDT0000 complete [ 4438.152175] LustreError: 140790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353584 with bad export cookie 15253207040893262608 [ 4438.155359] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4438.157545] LustreError: 140790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4438.300680] Lustre: server umount lustre-MDT0001 complete [ 4442.643491] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4443.828219] Key type lgssc unregistered [ 4443.970496] LNet: 141523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4443.974300] LNetError: 141523:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4443.984292] LNet: Removed LNI 192.168.203.154@tcp [ 4444.337132] Key type .llcrypt unregistered [ 4444.338970] Key type ._llcrypt unregistered [ 4451.664742] Key type ._llcrypt registered [ 4451.666253] Key type .llcrypt registered [ 4451.709192] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4458.324320] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4458.663107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4458.677579] alg: No test for adler32 (adler32-zlib) [ 4459.556830] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 4459.651894] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4461.239231] Key type lgssc registered [ 4461.651215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4465.202204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4467.857608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4470.060663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4472.349495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4474.817798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4474.837958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4474.843028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.935501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4475.948689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4475.993617] Lustre: lustre-MDT0000: new disk, initializing [ 4476.020317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.028788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4477.459575] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4482.072389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.109438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4482.146865] Lustre: 144672:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4482.181066] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4482.183912] Lustre: Skipped 1 previous similar message [ 4482.227239] Lustre: lustre-MDT0001: new disk, initializing [ 4482.269647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4482.288425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4482.293199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4483.918550] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4487.281529] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4488.879855] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4491.592662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.628933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4491.736328] Lustre: lustre-OST0000: new disk, initializing [ 4491.738556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4491.775299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.134359] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4497.561235] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4501.008759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4501.013210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4501.057110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4505.181644] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_module obdecho/obdecho [ 4505.935355] LustreError: 147297:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4505.939909] LustreError: 147297:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4506.765605] LustreError: 147393:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4511.201736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.202928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4511.206060] Lustre: Skipped 2 previous similar messages [ 4511.208019] Lustre: Skipped 1 previous similar message [ 4516.320762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.323274] Lustre: Skipped 3 previous similar messages [ 4516.674742] LustreError: 147732:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4516.714105] Lustre: server umount lustre-MDT0000 complete [ 4518.215236] LustreError: 144665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353664 with bad export cookie 4842125254020362504 [ 4518.218321] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.220358] LustreError: 144665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4518.303845] LustreError: 147934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4518.306951] LustreError: 147934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4518.385974] Lustre: server umount lustre-MDT0001 complete [ 4530.356853] LustreError: 148135:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4530.360299] LustreError: 148135:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4530.412091] Lustre: server umount lustre-OST0000 complete [ 4538.527916] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4542.220685] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4545.978944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.459649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4550.785768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.126441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.733755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.761089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.862610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4555.877930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4555.918490] Lustre: lustre-MDT0000: new disk, initializing [ 4555.950776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.959597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4557.445752] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4562.064042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4562.093690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4562.140119] Lustre: 151077:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4562.174119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4562.177033] Lustre: Skipped 1 previous similar message [ 4562.221445] Lustre: lustre-MDT0001: new disk, initializing [ 4562.269789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4562.284921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4562.289385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4563.881603] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4567.492419] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4568.776452] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4571.221868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.245400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4571.352726] Lustre: lustre-OST0000: new disk, initializing [ 4571.355699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4571.385562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4572.792424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4572.796514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4572.838924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4573.531905] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4577.105215] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4578.768930] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4578.856195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4580.394819] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4580.473592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4583.395457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4583.397077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.399519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.405168] Lustre: Skipped 1 previous similar message [ 4587.246083] LustreError: 154014:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4587.248745] LustreError: 154014:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4587.293540] Lustre: server umount lustre-OST0000 complete [ 4593.120306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4593.124739] LustreError: Skipped 1 previous similar message [ 4593.126845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.132872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.135856] Lustre: Skipped 1 previous similar message [ 4594.815779] LustreError: 154215:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4594.818439] LustreError: 154215:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4594.849092] Lustre: server umount lustre-MDT0000 complete [ 4596.181523] LustreError: 151068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353742 with bad export cookie 4842125254020363449 [ 4596.184882] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.187578] LustreError: 151068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4596.314607] Lustre: server umount lustre-MDT0001 complete [ 4597.536389] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:29:03 (1763353743) [ 4598.064902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4598.647432] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:29:05 (1763353745) [ 4601.369691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.570271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.974267] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4606.041028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4607.644283] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4609.586897] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4610.931182] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4613.381749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4615.509498] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4617.548678] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4632.035671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.036470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4632.037494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4632.037505] Lustre: Skipped 1 previous similar message [ 4632.040539] Lustre: Skipped 3 previous similar messages [ 4636.232689] LustreError: 157355:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4636.234908] LustreError: 157355:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4636.266843] Lustre: server umount lustre-MDT0000 complete [ 4637.152958] LustreError: 157255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.158372] LustreError: 157255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4637.559289] LustreError: 155071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353783 with bad export cookie 4842125254020364317 [ 4637.562467] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4637.564108] LustreError: 155071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4638.130481] LustreError: 155086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.272061] LustreError: 155086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.272316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.281446] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4642.283635] Lustre: Skipped 4 previous similar messages [ 4643.756668] Lustre: server umount lustre-MDT0001 complete [ 4645.938041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4650.339818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4650.429494] LustreError: 158617:0:(llog.c:1616:llog_backup()) MGC192.168.203.154@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4650.433124] Lustre: 158617:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.154@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4662.752122] LustreError: 158625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.757437] LustreError: 158625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4667.872382] LustreError: 158625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.088106] LustreError: 158641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.091974] LustreError: 158641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4694.450612] LustreError: 158623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.457512] LustreError: 158623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 4711.903396] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4711.906707] Lustre: 158617:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4727.264072] LustreError: 158643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.270882] LustreError: 158643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 4742.623329] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4742.625390] Lustre: 155081:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4752.871072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4752.874410] Lustre: Skipped 2 previous similar messages [ 4754.376743] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4756.963312] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4757.800469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.107428] LustreError: 155081:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4782.564226] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4332af438bd5117d [ 4782.568066] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 4784.241219] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4786.320858] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4786.996324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4787.686402] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4788.356574] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4791.031482] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4797.921918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.924181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4797.924790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4797.924796] Lustre: Skipped 1 previous similar message [ 4797.928149] Lustre: Skipped 1 previous similar message [ 4803.199848] LustreError: 160103:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4803.203110] LustreError: 160103:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4803.235763] Lustre: server umount lustre-MDT0000 complete [ 4804.623239] LustreError: 155071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353950 with bad export cookie 4842125254020370813 [ 4804.627145] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4804.627627] LustreError: 155071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4804.770902] Lustre: server umount lustre-MDT0001 complete [ 4816.542886] Lustre: server umount lustre-OST0000 complete [ 4823.767662] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4827.167939] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4830.644644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4832.983044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4840.383908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4842.696649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4845.412245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4845.436344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.523973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4845.536913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4845.574905] Lustre: lustre-MDT0000: new disk, initializing [ 4845.606140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.609477] Lustre: Skipped 1 previous similar message [ 4845.617341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4847.064868] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4851.612574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4851.642055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4851.681656] Lustre: 163448:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4851.791198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4853.278661] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4856.575645] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4857.937967] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4860.568659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.597800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4862.864570] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4866.347470] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4867.585038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4867.627179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4867.999956] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4868.089905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4869.591231] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4869.679772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4872.673913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4872.674103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.677205] LustreError: Skipped 1 previous similar message [ 4872.682995] Lustre: Skipped 1 previous similar message [ 4872.685731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4872.690127] Lustre: Skipped 6 previous similar messages [ 4876.463625] LustreError: 166385:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4876.465755] LustreError: 166385:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4876.507664] Lustre: server umount lustre-OST0000 complete [ 4882.399966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4892.127118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4892.204628] Lustre: server umount lustre-MDT0000 complete [ 4893.153237] LustreError: 165744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.158381] LustreError: 165744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 4893.437189] LustreError: 163439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354039 with bad export cookie 4842125254020387606 [ 4893.440057] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.442250] LustreError: 163439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4893.563903] Lustre: server umount lustre-MDT0001 complete [ 4896.551419] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:34:03 (1763354043) [ 4898.879451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.421936] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4903.276655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4904.696901] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4906.437577] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4907.662767] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4909.957332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4911.969194] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4913.712277] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4933.203897] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:34:39 (1763354079) [ 4934.624501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4934.624608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.631591] Lustre: Skipped 3 previous similar messages [ 4939.744601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.746984] Lustre: Skipped 6 previous similar messages [ 4940.298674] Lustre: server umount lustre-OST0000 complete [ 4947.710291] LustreError: 170126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4947.713140] LustreError: 170126:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4947.742598] Lustre: server umount lustre-MDT0000 complete [ 4949.072364] LustreError: 167441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354095 with bad export cookie 4842125254020388474 [ 4949.076496] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.077875] LustreError: 167441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4949.196060] Lustre: server umount lustre-MDT0001 complete [ 4952.900323] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4953.912793] Key type lgssc unregistered [ 4954.047381] LNet: 170860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4954.050502] LNetError: 170860:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4954.060345] LNet: Removed LNI 192.168.203.154@tcp [ 4954.377879] Key type .llcrypt unregistered [ 4954.379068] Key type ._llcrypt unregistered [ 4962.324656] Key type ._llcrypt registered [ 4962.325693] Key type .llcrypt registered [ 4962.363001] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4962.690880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4962.728820] alg: No test for adler32 (adler32-zlib) [ 4963.598058] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 4963.685298] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4965.271180] Key type lgssc registered [ 4965.632667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4968.328103] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4968.338297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4983.775282] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4988.895193] LustreError: 172059:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 4994.015384] LustreError: 172059:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4994.019295] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4994.021507] LustreError: MGC192.168.203.154@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4994.025728] LustreError: 172059:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4994.029438] LustreError: 172059:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 4994.032745] LustreError: 172059:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4994.035479] LustreError: 172059:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4994.056608] Lustre: server umount lustre-OST0000 complete [ 4994.058364] LustreError: 172059:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4996.955136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.108157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.453770] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5002.377105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5002.510787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5003.790106] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5005.667094] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5007.058135] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5009.551196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5009.641894] LustreError: 174150:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5009.646663] Lustre: 174150:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.154@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5009.704937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.790358] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5013.767922] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5014.556761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.557821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5014.561707] Lustre: Skipped 2 previous similar messages [ 5014.564282] Lustre: Skipped 1 previous similar message [ 5019.616725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5019.619027] Lustre: Skipped 3 previous similar messages [ 5020.667669] LustreError: 174801:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5020.670022] LustreError: 174801:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5020.694762] Lustre: server umount lustre-MDT0000 complete [ 5021.964169] LustreError: 173278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354168 with bad export cookie 2037518653514940652 [ 5021.968584] LustreError: 173278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5021.970449] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.040842] LustreError: 175004:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5022.043547] LustreError: 175004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5022.091721] Lustre: server umount lustre-MDT0001 complete [ 5033.521393] LustreError: 175205:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5033.525970] LustreError: 175205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5033.576283] Lustre: server umount lustre-OST0000 complete [ 5036.726237] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5039.968641] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5043.184668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.417464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5047.483017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5049.555546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5051.828827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5051.853046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.932727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5051.946360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5051.982727] Lustre: lustre-MDT0000: new disk, initializing [ 5052.009205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.018050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5053.336690] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5057.361504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5057.386761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5057.414032] Lustre: 177538:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.439060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5057.440635] Lustre: Skipped 1 previous similar message [ 5057.472918] Lustre: lustre-MDT0001: new disk, initializing [ 5057.506381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.517260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5057.520845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5058.679372] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5061.662069] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5062.833553] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5065.075748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5065.100524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5065.200960] Lustre: lustre-OST0000: new disk, initializing [ 5065.203891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5067.125808] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5067.575318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5067.579977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5067.613938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5070.232355] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5071.566332] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5071.635037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5072.954577] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5073.031220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5077.989422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5077.990269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.992552] LustreError: Skipped 1 previous similar message [ 5077.998851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5078.002410] Lustre: Skipped 1 previous similar message [ 5079.724071] LustreError: 180476:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5079.727151] LustreError: 180476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5079.763440] Lustre: server umount lustre-OST0000 complete [ 5083.104523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.104614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5083.105320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5083.108386] Lustre: Skipped 2 previous similar messages [ 5088.224064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5088.227024] Lustre: Skipped 2 previous similar messages [ 5095.391120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.423159] LustreError: 180677:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5095.425558] LustreError: 180677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5095.455165] Lustre: server umount lustre-MDT0000 complete [ 5096.715242] LustreError: 177528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354242 with bad export cookie 2037518653514941338 [ 5096.718116] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.720069] LustreError: 177528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5096.840239] Lustre: server umount lustre-MDT0001 complete [ 5099.850215] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:37:26 (1763354246) [ 5105.631302] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5106.709037] Key type lgssc unregistered [ 5106.845389] LNet: 182061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5106.848470] LNetError: 182061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5106.855294] LNet: Removed LNI 192.168.203.154@tcp [ 5107.169981] Key type .llcrypt unregistered [ 5107.171528] Key type ._llcrypt unregistered [ 5114.408269] Key type ._llcrypt registered [ 5114.409814] Key type .llcrypt registered [ 5114.452600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5114.781346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5114.820425] alg: No test for adler32 (adler32-zlib) [ 5115.677980] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5115.763780] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5117.359175] Key type lgssc registered [ 5117.736206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5120.456140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5120.463602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.641147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.914725] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5125.790159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5125.910861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5127.271309] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5129.005760] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5130.132064] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5132.644464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5132.713179] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5132.714329] LustreError: 184874:0:(llog.c:1657:llog_backup()) MGC192.168.203.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5132.717112] Lustre: 184874:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.154@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5132.752143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.617440] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5136.339368] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5140.960937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.962877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5140.964697] Lustre: Skipped 2 previous similar messages [ 5140.968708] Lustre: Skipped 2 previous similar messages [ 5143.164524] LustreError: 185528:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5143.190887] Lustre: server umount lustre-MDT0000 complete [ 5144.389453] LustreError: 183261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354290 with bad export cookie 12597450516754448486 [ 5144.392560] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5144.394925] LustreError: 183261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5144.466090] LustreError: 185731:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5144.468632] LustreError: 185731:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5144.520983] Lustre: server umount lustre-MDT0001 complete [ 5154.927945] LustreError: 185932:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5154.930175] LustreError: 185932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5154.968558] Lustre: server umount lustre-OST0000 complete [ 5157.705678] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5160.965036] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5164.064041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5166.342894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5168.375085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5170.528839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5172.781930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5172.805546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.886687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5172.898426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5172.937858] Lustre: lustre-MDT0000: new disk, initializing [ 5172.971324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5172.978855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5174.400051] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5178.420984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.439794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5178.467648] Lustre: 188265:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5178.481804] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5178.483217] Lustre: Skipped 1 previous similar message [ 5178.484340] LustreError: 189029:0:(llog.c:1657:llog_backup()) MGC192.168.203.154@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5178.486636] LustreError: 189029:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5178.488491] Lustre: 189029:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.154@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5178.495862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5178.498147] Lustre: Skipped 1 previous similar message [ 5178.536165] Lustre: lustre-MDT0001: new disk, initializing [ 5178.571959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5178.583828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5178.587990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5179.803471] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5182.792126] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5183.950621] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5186.108405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5186.133819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5186.217549] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5186.218751] Lustre: Skipped 1 previous similar message [ 5186.220606] LustreError: 189985:0:(llog.c:1657:llog_backup()) MGC192.168.203.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5186.225642] LustreError: 189985:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5186.229157] Lustre: 189985:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.154@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5186.234285] Lustre: 189985:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5186.243947] Lustre: lustre-OST0000: new disk, initializing [ 5186.245738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5188.046721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5191.003243] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5192.186305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5192.189729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5192.233550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5192.410229] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5192.488442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5193.748225] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5193.819153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5197.281743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.282268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.282887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.285328] LustreError: Skipped 1 previous similar message [ 5197.289508] Lustre: Skipped 1 previous similar message [ 5200.493440] LustreError: 191202:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5200.496111] LustreError: 191202:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5200.526355] Lustre: server umount lustre-OST0000 complete [ 5202.401235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.402174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5202.404436] Lustre: Skipped 1 previous similar message [ 5202.406443] Lustre: Skipped 1 previous similar message [ 5207.520143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5207.521885] Lustre: Skipped 2 previous similar messages [ 5207.803776] LustreError: 191404:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5207.805674] LustreError: 191404:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5207.834313] Lustre: server umount lustre-MDT0000 complete [ 5209.059566] LustreError: 188256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354355 with bad export cookie 12597450516754449186 [ 5209.062271] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.064566] LustreError: 188256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5209.189822] Lustre: server umount lustre-MDT0001 complete [ 5211.839163] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:39:18 (1763354358) [ 5216.868574] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5220.125680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.269897] LustreError: 192969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.275161] LustreError: 192969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5220.286399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.288418] Lustre: Skipped 1 previous similar message [ 5221.559821] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5224.414874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5225.849413] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5226.860358] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5229.166209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5231.229680] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5234.153158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5234.174353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5234.233079] Lustre: lustre-OST0001: new disk, initializing [ 5234.235595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5236.316359] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5238.270437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5238.274753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5238.311820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5241.911782] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5247.694470] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5253.356891] Lustre: DEBUG MARKER: ADD OST3 [ 5255.107463] loop: module loaded [ 5255.125478] loop0: detected capacity change from 0 to 204800000 [ 5255.264170] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5255.269747] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5255.274476] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5255.326260] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5257.411819] loop0: detected capacity change from 0 to 204800000 [ 5259.403976] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5259.430870] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5259.507877] Lustre: lustre-OST0003: new disk, initializing [ 5259.510712] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5259.557537] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5259.560882] Lustre: Skipped 3 previous similar messages [ 5260.292321] Lustre: DEBUG MARKER: STOP OST3 [ 5260.973969] Lustre: Failing over lustre-OST0003 [ 5261.001286] LustreError: 197284:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5261.004545] LustreError: 197284:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5261.021328] Lustre: server umount lustre-OST0003 complete [ 5261.191121] LustreError: 194424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.669064] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5264.889043] LustreError: 194423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.893430] LustreError: 194423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5265.476897] Lustre: DEBUG MARKER: START OST3 again [ 5267.797388] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5267.881340] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5269.311369] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5269.320358] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5269.618944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5269.623302] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5269.632157] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5269.810714] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5283.378792] Lustre: Failing over lustre-OST0003 [ 5283.401089] LustreError: 199258:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5283.403013] LustreError: 199258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5283.422403] Lustre: server umount lustre-OST0003 complete [ 5285.344219] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5285.344328] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.346884] LustreError: Skipped 1 previous similar message [ 5285.347585] LustreError: 194423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.350252] Lustre: Skipped 1 previous similar message [ 5285.358492] LustreError: 194423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5293.540134] LustreError: 194424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.545689] LustreError: 194424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5296.095877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5296.099346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.103386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.244649] Lustre: server umount lustre-MDT0000 complete [ 5303.537510] LustreError: 192948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354449 with bad export cookie 12597450516754450054 [ 5303.542437] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.543329] LustreError: 192948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5303.776194] LustreError: 192963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5303.780939] LustreError: 192963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5308.896615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5308.898880] Lustre: Skipped 7 previous similar messages [ 5309.868361] Lustre: server umount lustre-MDT0001 complete [ 5311.136830] Lustre: server umount lustre-OST0000 complete [ 5312.381049] Lustre: server umount lustre-OST0001 complete [ 5314.317068] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5316.695237] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5320.108238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5322.150383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5324.016665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5326.115198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5328.392253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5328.415692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.505714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5328.519377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5328.562242] Lustre: lustre-MDT0000: new disk, initializing [ 5328.589120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.591202] Lustre: Skipped 1 previous similar message [ 5328.596853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5329.959973] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5334.237340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5334.260857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5334.301109] Lustre: 202314:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5334.415887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5335.845373] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5339.122672] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5340.450497] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5342.860530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5342.884402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5344.506509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5344.545777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5344.906329] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5347.897761] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5349.170628] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5349.238831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5350.513353] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5350.593995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5354.977910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5354.978024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.985951] Lustre: Skipped 5 previous similar messages [ 5354.988388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.990434] Lustre: Skipped 1 previous similar message [ 5357.294109] LustreError: 205250:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5357.297223] LustreError: 205250:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 5357.344723] Lustre: server umount lustre-OST0000 complete [ 5360.096786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5360.099643] LustreError: Skipped 1 previous similar message [ 5364.696603] Lustre: server umount lustre-MDT0000 complete [ 5365.215842] LustreError: 202325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.220220] LustreError: 202325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5365.883136] LustreError: 202306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354512 with bad export cookie 12597450516754452322 [ 5365.885869] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.886778] LustreError: 202306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5366.002918] Lustre: server umount lustre-MDT0001 complete [ 5367.135122] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:41:53 (1763354513) [ 5372.122826] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5374.880064] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5377.865555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5379.825676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5381.698023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5383.687951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5385.798675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5385.819615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.893960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5385.903804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5385.905568] Lustre: Skipped 3 previous similar messages [ 5385.941377] Lustre: lustre-MDT0000: new disk, initializing [ 5385.942866] Lustre: Skipped 2 previous similar messages [ 5385.971725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5385.975078] Lustre: Skipped 2 previous similar messages [ 5387.173236] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5391.055516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.080351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5391.112603] Lustre: 208491:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5391.218543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5392.498655] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5395.523903] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5396.701889] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5399.055053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.080220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5399.207121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.210394] Lustre: Skipped 4 previous similar messages [ 5400.285025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5401.096674] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5404.256883] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5405.634299] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5405.703251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5407.033095] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5407.108297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5409.870399] Lustre: server umount lustre-OST0000 complete [ 5411.807971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5411.811541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.815827] Lustre: Skipped 3 previous similar messages [ 5411.817528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5411.820363] Lustre: Skipped 2 previous similar messages [ 5421.050924] Lustre: server umount lustre-MDT0000 complete [ 5422.364060] LustreError: 208484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354568 with bad export cookie 12597450516754453204 [ 5422.368172] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.370151] LustreError: 208484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5431.779515] LustreError: 211836:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5431.781419] LustreError: 211836:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 5431.833781] Lustre: server umount lustre-MDT0001 complete [ 5434.782303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.947405] LustreError: 212313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.379730] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5439.264670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5440.677249] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5442.405566] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5443.587041] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5445.853300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5447.941490] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5449.752330] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5469.663905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.666592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.670299] Lustre: Skipped 1 previous similar message [ 5472.658640] Lustre: server umount lustre-OST0000 complete [ 5480.089695] Lustre: server umount lustre-MDT0000 complete [ 5481.384510] LustreError: 212293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354627 with bad export cookie 12597450516754454086 [ 5481.388565] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.388769] LustreError: 212293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5481.510526] Lustre: server umount lustre-MDT0001 complete [ 5585.901496] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5587.034179] Key type lgssc unregistered [ 5587.178361] LNet: 215646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5587.180668] LNetError: 215646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5587.190430] LNet: Removed LNI 192.168.203.154@tcp [ 5587.507107] Key type .llcrypt unregistered [ 5587.508210] Key type ._llcrypt unregistered [ 5596.028485] Key type ._llcrypt registered [ 5596.030074] Key type .llcrypt registered [ 5596.068807] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5602.374216] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5602.724080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5602.780861] alg: No test for adler32 (adler32-zlib) [ 5603.641645] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5603.723928] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5605.303161] Key type lgssc registered [ 5605.684604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5608.607312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5610.765663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.846364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5615.088493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5617.541430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5617.562152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5617.566455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.667857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5618.680132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5618.719040] Lustre: lustre-MDT0000: new disk, initializing [ 5618.747433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.754798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5620.103861] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5624.321403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5624.343167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5624.376978] Lustre: 218954:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5624.403275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5624.405356] Lustre: Skipped 1 previous similar message [ 5624.449831] Lustre: lustre-MDT0001: new disk, initializing [ 5624.484725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5624.497112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5624.501719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5625.930156] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5629.193241] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5630.518088] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5632.961133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5632.989824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5633.095299] Lustre: lustre-OST0000: new disk, initializing [ 5633.098551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5633.130122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.183361] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5636.343068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5636.346167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5636.378514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5638.373593] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5639.770548] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5639.847942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5641.228713] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5641.307547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5644.012244] LustreError: 221893:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5644.052272] Lustre: server umount lustre-OST0000 complete [ 5646.816693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.817478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5646.821779] Lustre: Skipped 1 previous similar message [ 5651.936378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5651.939452] Lustre: Skipped 2 previous similar messages [ 5655.010908] LustreError: 222094:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5655.012867] LustreError: 222094:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5655.038430] Lustre: server umount lustre-MDT0000 complete [ 5656.332309] LustreError: 218945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354802 with bad export cookie 16584321013198362313 [ 5656.335880] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5656.336455] LustreError: 218945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.789424] LustreError: 222296:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5666.793252] LustreError: 222296:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5666.859241] Lustre: server umount lustre-MDT0001 complete [ 5673.348822] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5676.949416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.117796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.462496] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5681.378461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5681.511509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5682.882279] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5683.910466] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5686.137550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5688.108350] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5691.070580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5691.091903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.153945] Lustre: lustre-OST0001: new disk, initializing [ 5691.157425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5691.195755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5691.197931] Lustre: Skipped 1 previous similar message [ 5692.487210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5692.492626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5692.525449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5693.298274] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5698.814436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5703.398393] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5712.349305] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:47:38 (1763354858) [ 5713.378387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.381265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.381963] Lustre: Skipped 2 previous similar messages [ 5713.383712] Lustre: Skipped 1 previous similar message [ 5718.497806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.500093] Lustre: Skipped 5 previous similar messages [ 5719.169625] LustreError: 226847:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5719.171991] LustreError: 226847:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5719.205715] Lustre: server umount lustre-MDT0000 complete [ 5720.471307] LustreError: 223451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354866 with bad export cookie 16584321013198363181 [ 5720.474946] LustreError: 223451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5720.475866] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5720.594615] Lustre: server umount lustre-MDT0001 complete [ 5731.826692] LustreError: 227250:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5731.828598] LustreError: 227250:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5731.846559] Lustre: server umount lustre-OST0000 complete [ 5743.348134] LustreError: 227452:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5743.350843] LustreError: 227452:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5743.403794] Lustre: server umount lustre-OST0001 complete [ 5745.415847] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5748.867708] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5752.275125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5754.389872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5756.381826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5758.610060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5760.980403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5761.004552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.114050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5761.129268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5761.175472] Lustre: lustre-MDT0000: new disk, initializing [ 5761.203049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.211926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5762.699180] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5766.759736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5766.777656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5766.807918] Lustre: 229633:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5766.903143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5768.139976] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5771.204453] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5772.493536] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5774.874474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5774.899856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5775.017445] Lustre: lustre-OST0000: new disk, initializing [ 5775.019213] Lustre: Skipped 1 previous similar message [ 5775.021498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5775.023665] Lustre: Skipped 2 previous similar messages [ 5776.056873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5776.060443] Lustre: Skipped 1 previous similar message [ 5776.062477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5776.093844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5777.091284] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5780.352917] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5781.894893] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5781.981456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5783.491113] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5783.582025] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5786.286143] LustreError: 232569:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5786.288956] LustreError: 232569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5786.333459] Lustre: server umount lustre-OST0000 complete [ 5791.713610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.714445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.717816] Lustre: Skipped 2 previous similar messages [ 5791.722634] Lustre: Skipped 1 previous similar message [ 5796.857850] Lustre: server umount lustre-MDT0000 complete [ 5798.211675] LustreError: 230415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354944 with bad export cookie 16584321013198365281 [ 5798.215696] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5798.217446] LustreError: 230415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5808.658033] Lustre: server umount lustre-MDT0001 complete [ 5811.475325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.630330] LustreError: 233451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.636812] LustreError: 233451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5811.653067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.655606] Lustre: Skipped 2 previous similar messages [ 5813.030413] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5815.999680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5817.547808] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5819.412831] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5820.670803] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5823.009549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.146736] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5827.031038] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5975.011799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.016161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.016717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.018773] LustreError: Skipped 1 previous similar message [ 5975.021664] Lustre: Skipped 3 previous similar messages [ 5986.271123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5986.292174] LustreError: 235737:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5986.295151] LustreError: 235737:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5986.334095] Lustre: server umount lustre-OST0000 complete [ 5990.369190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5990.369513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.377496] Lustre: Skipped 2 previous similar messages [ 5993.753293] Lustre: server umount lustre-MDT0000 complete [ 5994.978402] LustreError: 233477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355141 with bad export cookie 16584321013198366163 [ 5994.982212] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5994.983900] LustreError: 233477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5995.094881] Lustre: server umount lustre-MDT0001 complete [ 6089.077633] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6090.115268] Key type lgssc unregistered [ 6090.250475] LNet: 236706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6090.252972] LNetError: 236706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6090.262403] LNet: Removed LNI 192.168.203.154@tcp [ 6090.587121] Key type .llcrypt unregistered [ 6090.588561] Key type ._llcrypt unregistered [ 6099.090046] Key type ._llcrypt registered [ 6099.090985] Key type .llcrypt registered [ 6099.129154] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6105.514761] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6105.918650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6105.971378] alg: No test for adler32 (adler32-zlib) [ 6106.828496] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6106.905262] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6108.487127] Key type lgssc registered [ 6108.905297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.971275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6114.118908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6116.121450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6118.268503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6123.692843] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6127.088213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6127.106169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6127.110339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.191631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6128.203546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6128.249207] Lustre: lustre-MDT0000: new disk, initializing [ 6128.276605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.286361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6129.609031] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6133.797874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6133.824228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6133.866861] Lustre: 240702:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6133.894173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6133.897049] Lustre: Skipped 1 previous similar message [ 6133.942087] Lustre: lustre-MDT0001: new disk, initializing [ 6133.981928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6133.997130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6134.001940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6135.355283] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6137.684735] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6139.942503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6139.961462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6140.063046] Lustre: lustre-OST0000: new disk, initializing [ 6140.066073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6140.102526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6141.891277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6141.894817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6141.956077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6142.164504] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6146.318253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6146.343863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6146.403353] Lustre: lustre-OST0001: new disk, initializing [ 6146.405882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6146.439406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.270877] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6151.164370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6151.167826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6151.193154] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6153.541897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6156.030311] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6164.959982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6164.962256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.967763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6166.499352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.499908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6166.503651] Lustre: Skipped 2 previous similar messages [ 6166.505995] Lustre: Skipped 2 previous similar messages [ 6169.994308] LustreError: 244231:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6170.030646] Lustre: server umount lustre-MDT0000 complete [ 6171.313498] LustreError: 241483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355317 with bad export cookie 8772665633756361150 [ 6171.317124] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6171.319351] LustreError: 241483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6171.387246] LustreError: 244432:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6171.390788] LustreError: 244432:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6171.454723] Lustre: server umount lustre-MDT0001 complete [ 6183.031101] LustreError: 244632:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6183.033363] LustreError: 244632:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6183.052691] Lustre: server umount lustre-OST0000 complete [ 6194.612655] LustreError: 244834:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6194.614738] LustreError: 244834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6194.661653] Lustre: server umount lustre-OST0001 complete [ 6199.558779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6202.365877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6202.692799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6205.446023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6208.147022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6208.462819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6214.699691] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6218.319309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.325657] Lustre: lustre-MDT0000: reset Object Index mappings [ 6218.516429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.812027] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6222.607847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6222.614837] Lustre: lustre-MDT0001: reset Object Index mappings [ 6224.105753] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6225.062579] Lustre: 249250:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6227.136061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6227.262789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.265278] Lustre: Skipped 1 previous similar message [ 6228.325973] LustreError: 249606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.330709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6228.331525] LustreError: 249606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6229.072139] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6231.739930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6232.874230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6233.585196] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6236.531358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6237.647381] Lustre: 251097:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6248.419314] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.419327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6248.421249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6248.421258] Lustre: Skipped 1 previous similar message [ 6248.423524] Lustre: Skipped 3 previous similar messages [ 6249.982280] LustreError: 251201:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6249.984424] LustreError: 251201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6250.014291] Lustre: server umount lustre-MDT0000 complete [ 6251.268953] LustreError: 248123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355397 with bad export cookie 8772665633756367926 [ 6251.273854] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6251.275059] LustreError: 248123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6251.409724] Lustre: server umount lustre-MDT0001 complete [ 6262.711110] LustreError: 251603:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6262.713461] LustreError: 251603:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6262.732095] Lustre: server umount lustre-OST0000 complete [ 6274.267598] Lustre: server umount lustre-OST0001 complete [ 6280.881153] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:57:07 (1763355427) [ 6285.979437] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6289.123760] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6292.277203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6294.448681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6296.478555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6298.648834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6300.872206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6300.892933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.970769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6300.981749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6301.015864] Lustre: lustre-MDT0000: new disk, initializing [ 6301.039255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.042090] Lustre: Skipped 1 previous similar message [ 6301.048383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6302.363901] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6306.572471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6306.594155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6306.626600] Lustre: 255321:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6306.717891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6307.926797] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6311.068303] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6312.328515] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6314.665359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6314.689419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6314.804385] Lustre: lustre-OST0000: new disk, initializing [ 6314.805937] Lustre: Skipped 1 previous similar message [ 6314.807786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6314.810491] Lustre: Skipped 2 previous similar messages [ 6316.409785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6316.412948] Lustre: Skipped 1 previous similar message [ 6316.415165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6316.454896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6316.883424] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6320.075837] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6321.469982] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6321.565910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6322.967700] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6323.052433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6326.753361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6326.754020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.756373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6326.756378] Lustre: Skipped 3 previous similar messages [ 6326.757107] LustreError: Skipped 1 previous similar message [ 6326.766726] Lustre: Skipped 1 previous similar message [ 6329.772865] LustreError: 258257:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6329.775224] LustreError: 258257:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6329.815194] Lustre: server umount lustre-OST0000 complete [ 6331.872981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.873663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6331.876499] Lustre: Skipped 1 previous similar message [ 6331.878117] Lustre: Skipped 1 previous similar message [ 6337.239190] Lustre: server umount lustre-MDT0000 complete [ 6338.536206] LustreError: 255313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355484 with bad export cookie 8772665633756370019 [ 6338.540235] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.540907] LustreError: 255313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6338.651139] Lustre: server umount lustre-MDT0001 complete [ 6345.092404] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6348.737861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.886348] LustreError: 259834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.901504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6348.903697] Lustre: Skipped 2 previous similar messages [ 6350.286879] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6353.267667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6354.815916] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6355.859118] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6358.162036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6360.355347] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6363.267950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.295346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6363.370619] Lustre: lustre-OST0001: new disk, initializing [ 6363.373669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6365.268551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6365.274243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6365.316765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6365.453288] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6370.025644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6371.319963] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6378.217461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6378.232692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6381.027291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.029049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6381.033385] Lustre: Skipped 3 previous similar messages [ 6381.035336] Lustre: Skipped 5 previous similar messages [ 6386.046141] LustreError: 263340:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6386.048662] LustreError: 263340:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6386.079597] Lustre: server umount lustre-MDT0000 complete [ 6386.144510] LustreError: 259834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.149386] LustreError: 259834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6387.339091] LustreError: 262096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355533 with bad export cookie 8772665633756370901 [ 6387.342673] LustreError: 262096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6387.342822] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6387.453026] Lustre: server umount lustre-MDT0001 complete [ 6399.492016] Lustre: server umount lustre-OST0000 complete [ 6410.008659] Lustre: server umount lustre-OST0001 complete [ 6415.641648] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6419.334466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.496121] LustreError: 265119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.518000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6419.520720] Lustre: Skipped 3 previous similar messages [ 6420.993367] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6423.974163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6425.469346] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6426.496035] Lustre: 266229:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6428.804523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6429.985959] LustreError: 266585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.991208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6430.997116] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6433.947279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6435.060977] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6436.063540] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6439.359104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6440.594203] Lustre: 268081:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6447.452991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6447.458753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6450.799753] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:59:57 (1763355597) [ 6464.992209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6464.994664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.998544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6468.548240] LustreError: 268793:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6468.550389] LustreError: 268793:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6468.583318] Lustre: server umount lustre-MDT0000 complete [ 6469.851443] LustreError: 265100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355616 with bad export cookie 8772665633756377180 [ 6469.853654] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6469.856113] LustreError: 265100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6471.136268] LustreError: 268051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.142313] LustreError: 268051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6476.027947] Lustre: server umount lustre-MDT0001 complete [ 6483.461380] Lustre: server umount lustre-OST0000 complete [ 6490.845950] Lustre: server umount lustre-OST0001 complete [ 6493.008451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6496.631586] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6500.373452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6502.653233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6504.887284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6507.202354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6512.815354] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6516.591284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6516.617812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.718674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6516.735477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6516.780949] Lustre: lustre-MDT0000: new disk, initializing [ 6516.816102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6518.299509] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6522.626708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6522.653057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6522.696073] Lustre: 272287:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6522.833455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6524.263101] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6528.828546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6528.856360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6529.265969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6531.052647] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6535.254247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.285058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6537.202777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6537.374407] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6542.005826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6552.747979] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 00:01:39 (1763355699) [ 6553.567988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.571069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.574560] Lustre: Skipped 5 previous similar messages [ 6553.576360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.578095] Lustre: Skipped 8 previous similar messages [ 6559.648133] Lustre: server umount lustre-MDT0000 complete [ 6560.863187] LustreError: 272278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355707 with bad export cookie 8772665633756385041 [ 6560.866228] LustreError: 272278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6560.866573] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6560.983754] Lustre: server umount lustre-MDT0001 complete [ 6571.974767] Lustre: server umount lustre-OST0000 complete [ 6583.516654] Lustre: server umount lustre-OST0001 complete [ 6589.508567] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6592.705303] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6596.019087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.275044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6600.377793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6602.615878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.252544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.282645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.384322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6605.387187] Lustre: Skipped 2 previous similar messages [ 6605.398732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6605.401617] Lustre: Skipped 4 previous similar messages [ 6605.451829] Lustre: lustre-MDT0000: new disk, initializing [ 6605.453954] Lustre: Skipped 3 previous similar messages [ 6605.483092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.485211] Lustre: Skipped 7 previous similar messages [ 6605.491350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6605.493707] Lustre: Skipped 3 previous similar messages [ 6606.930199] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6611.071057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6611.094450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6611.129736] Lustre: 279343:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6611.245583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6611.247911] Lustre: Skipped 2 previous similar messages [ 6612.595738] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6615.708978] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6616.908057] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6619.143282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6619.169610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6620.390311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6621.284208] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6624.388266] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6625.095972] Lustre: Mounted lustre-client [ 6625.403497] LustreError: 281844:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9954cf8b1800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6625.410320] LustreError: 281844:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6625.411845] LustreError: 281844:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 6625.430137] Lustre: Unmounted lustre-client [ 6630.370610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.374173] Lustre: Skipped 3 previous similar messages [ 6630.376087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6630.377857] Lustre: Skipped 5 previous similar messages [ 6634.524659] Lustre: server umount lustre-OST0000 complete [ 6637.024106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6650.335182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6650.398935] Lustre: server umount lustre-MDT0000 complete [ 6651.644597] LustreError: 280122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355797 with bad export cookie 8772665633756386560 [ 6651.647790] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6651.649713] LustreError: 280122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6651.773130] Lustre: server umount lustre-MDT0001 complete [ 6655.501166] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6656.535531] Key type lgssc unregistered [ 6684.447168] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [ 6685.343449] LNet: 283122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6685.346818] LNetError: 283122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6756.005459] LNet: Removed LNI 192.168.203.154@tcp [ 6756.357128] Key type .llcrypt unregistered [ 6756.358689] Key type ._llcrypt unregistered [ 6762.295333] Key type ._llcrypt registered [ 6762.296266] Key type .llcrypt registered [ 6762.330976] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6768.783279] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6769.219403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6769.244025] alg: No test for adler32 (adler32-zlib) [ 6770.109228] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6770.196621] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6771.791160] Key type lgssc registered [ 6772.193542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6775.360709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6777.566848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6779.595830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.726490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6782.330918] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:05:28 (1763355928) [ 6784.606056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6784.629287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6784.635922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.717671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6785.728225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6785.762490] Lustre: lustre-MDT0000: new disk, initializing [ 6785.786162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.792499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6787.108446] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6791.283228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.308756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6791.343623] Lustre: 286291:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6791.374767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6791.376422] Lustre: Skipped 1 previous similar message [ 6791.410325] Lustre: lustre-MDT0001: new disk, initializing [ 6791.444907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.456675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6791.460731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6792.769830] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6795.920336] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6797.138395] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6799.422866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.453893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6799.556356] Lustre: lustre-OST0000: new disk, initializing [ 6799.558157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6799.589662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.593732] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6804.633887] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6809.084819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6809.087943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6809.126306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6814.177773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6814.178883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.179909] LustreError: Skipped 1 previous similar message [ 6814.185851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.187803] Lustre: Skipped 1 previous similar message [ 6815.788822] LustreError: 288732:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6815.824710] Lustre: server umount lustre-OST0000 complete [ 6817.248044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6817.250399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.253367] Lustre: Skipped 1 previous similar message [ 6817.254889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.135414] LustreError: 288935:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6819.138639] LustreError: 288935:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6819.170783] Lustre: server umount lustre-MDT0000 complete [ 6819.296942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.433584] LustreError: 286283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355966 with bad export cookie 8217493543124861456 [ 6820.436804] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6820.441223] LustreError: 286283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6820.506502] LustreError: 289137:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6820.508443] LustreError: 289137:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6820.565927] Lustre: server umount lustre-MDT0001 complete [ 6824.322831] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6825.418709] Key type lgssc unregistered [ 6825.554437] LNet: 289667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6825.556637] LNetError: 289667:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6825.569356] LNet: Removed LNI 192.168.203.154@tcp [ 6825.871031] Key type .llcrypt unregistered [ 6825.872303] Key type ._llcrypt unregistered [ 6859.204257] Key type ._llcrypt registered [ 6859.205701] Key type .llcrypt registered [ 6859.242886] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6859.349298] Key type .llcrypt unregistered [ 6859.350379] Key type ._llcrypt unregistered [ 6887.006016] Key type ._llcrypt registered [ 6887.007094] Key type .llcrypt registered [ 6887.049490] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6887.158237] Key type .llcrypt unregistered [ 6887.159259] Key type ._llcrypt unregistered [ 6898.398810] Key type ._llcrypt registered [ 6898.400483] Key type .llcrypt registered [ 6898.444825] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6898.550348] Key type .llcrypt unregistered [ 6898.551258] Key type ._llcrypt unregistered [ 6909.788251] Key type ._llcrypt registered [ 6909.789308] Key type .llcrypt registered [ 6909.824343] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6909.914310] Key type .llcrypt unregistered [ 6909.915412] Key type ._llcrypt unregistered [ 6937.356042] Key type ._llcrypt registered [ 6937.357112] Key type .llcrypt registered [ 6937.391868] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6937.494370] Key type .llcrypt unregistered [ 6937.495878] Key type ._llcrypt unregistered [ 6948.299389] Key type ._llcrypt registered [ 6948.300724] Key type .llcrypt registered [ 6948.337185] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6948.427194] Key type .llcrypt unregistered [ 6948.428334] Key type ._llcrypt unregistered [ 6959.735379] Key type ._llcrypt registered [ 6959.736433] Key type .llcrypt registered [ 6959.774023] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6959.879240] Key type .llcrypt unregistered [ 6959.880936] Key type ._llcrypt unregistered [ 6971.331607] Key type ._llcrypt registered [ 6971.333550] Key type .llcrypt registered [ 6971.372232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6971.478433] Key type .llcrypt unregistered [ 6971.479728] Key type ._llcrypt unregistered [ 6982.962823] Key type ._llcrypt registered [ 6982.964116] Key type .llcrypt registered [ 6983.001586] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6983.104178] Key type .llcrypt unregistered [ 6983.105494] Key type ._llcrypt unregistered [ 6994.644367] Key type ._llcrypt registered [ 6994.646033] Key type .llcrypt registered [ 6994.687284] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6994.801474] Key type .llcrypt unregistered [ 6994.803346] Key type ._llcrypt unregistered [ 7015.594172] Key type ._llcrypt registered [ 7015.595140] Key type .llcrypt registered [ 7015.627679] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7015.717346] Key type .llcrypt unregistered [ 7015.718357] Key type ._llcrypt unregistered [ 7026.327847] Key type ._llcrypt registered [ 7026.328815] Key type .llcrypt registered [ 7026.368251] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7026.467173] Key type .llcrypt unregistered [ 7026.468190] Key type ._llcrypt unregistered [ 7036.830952] Key type ._llcrypt registered [ 7036.832046] Key type .llcrypt registered [ 7036.863224] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7036.950180] Key type .llcrypt unregistered [ 7036.951151] Key type ._llcrypt unregistered [ 7056.840854] Key type ._llcrypt registered [ 7056.842529] Key type .llcrypt registered [ 7056.890836] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 7064.218470] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7064.597447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7064.615171] alg: No test for adler32 (adler32-zlib) [ 7065.486829] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7065.599891] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7067.199176] Key type lgssc registered [ 7067.712145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7071.094337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7073.355077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7075.476051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7077.748720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7078.331948] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6779 sec ========== 00:10:24 (1763356224) [ 7078.913599] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:10:25 (1763356225) === [ 7080.017377] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:10:26 (1763356226) === [ 7092.974173] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7093.992728] Key type lgssc unregistered [ 7094.117445] LNet: 311377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7094.119632] LNetError: 311377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7094.127434] LNet: Removed LNI 192.168.203.154@tcp [ 7094.442375] Key type .llcrypt unregistered [ 7094.443676] Key type ._llcrypt unregistered