[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493085750 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002369] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006910] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010031] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.020679] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024547] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] *** VALIDATE tmpfs *** [ 0.029239] *** VALIDATE proc *** [ 0.030214] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032241] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034089] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040178] debug: unmapping init [mem 0xffffffffbde59000-0xffffffffbde60fff] [ 0.042159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043623] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046020] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051266] rcu: Hierarchical SRCU implementation. [ 0.053417] smp: Bringing up secondary CPUs ... [ 0.054549] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.058362] smp: Brought up 1 node, 4 CPUs [ 0.060010] smpboot: Max logical packages: 1 [ 0.061017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131670] node 0 deferred pages initialised in 66ms [ 0.134057] devtmpfs: initialized [ 0.135209] x86/mm: Memory block size: 128MB [ 0.137654] gcov: version magic: 0x41383552 [ 0.139050] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141227] pinctrl core: initialized pinctrl subsystem [ 0.142191] [ 0.142818] ************************************************************* [ 0.143013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144014] ** ** [ 0.145010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146007] ** ** [ 0.147012] ** This means that this kernel is built to expose internal ** [ 0.148013] ** IOMMU data structures, which may compromise security on ** [ 0.149023] ** your system. ** [ 0.150012] ** ** [ 0.151013] ** If you see this message and you are not debugging the ** [ 0.152011] ** kernel, report this immediately to your vendor! ** [ 0.153009] ** ** [ 0.154013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155012] ************************************************************* [ 0.156747] NET: Registered protocol family 16 [ 0.157420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160562] cpuidle: using governor menu [ 0.162517] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163442] PCI: Using configuration type 1 for base access [ 0.164120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.171147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.172040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174035] cryptd: max_cpu_qlen set to 1000 [ 0.176237] ACPI: Added _OSI(Module Device) [ 0.177012] ACPI: Added _OSI(Processor Device) [ 0.178012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.179011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.183118] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187477] ACPI: Interpreter enabled [ 0.188069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189011] ACPI: Using IOAPIC for interrupt routing [ 0.190089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.191485] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.201560] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.202041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.203019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.204063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.206316] acpiphp: Slot [2] registered [ 0.207108] acpiphp: Slot [5] registered [ 0.208101] acpiphp: Slot [6] registered [ 0.210115] acpiphp: Slot [7] registered [ 0.211103] acpiphp: Slot [8] registered [ 0.213097] acpiphp: Slot [9] registered [ 0.214101] acpiphp: Slot [10] registered [ 0.216109] acpiphp: Slot [3] registered [ 0.217113] acpiphp: Slot [4] registered [ 0.219102] acpiphp: Slot [11] registered [ 0.220089] acpiphp: Slot [12] registered [ 0.222103] acpiphp: Slot [13] registered [ 0.224096] acpiphp: Slot [14] registered [ 0.225093] acpiphp: Slot [15] registered [ 0.227117] acpiphp: Slot [16] registered [ 0.229102] acpiphp: Slot [17] registered [ 0.230140] acpiphp: Slot [18] registered [ 0.232106] acpiphp: Slot [19] registered [ 0.234108] acpiphp: Slot [20] registered [ 0.235095] acpiphp: Slot [21] registered [ 0.237098] acpiphp: Slot [22] registered [ 0.238115] acpiphp: Slot [23] registered [ 0.240094] acpiphp: Slot [24] registered [ 0.242096] acpiphp: Slot [25] registered [ 0.243092] acpiphp: Slot [26] registered [ 0.245158] acpiphp: Slot [27] registered [ 0.247102] acpiphp: Slot [28] registered [ 0.248107] acpiphp: Slot [29] registered [ 0.250101] acpiphp: Slot [30] registered [ 0.252092] acpiphp: Slot [31] registered [ 0.253092] PCI host bridge to bus 0000:00 [ 0.255019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.257021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.260082] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.263023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.266022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.268022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.270166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.273645] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.277468] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.287761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.293573] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.296018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.301018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304540] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.308122] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.309040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.311765] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.317013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.331018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.336923] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.344126] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.354013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.363015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.381021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.390281] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.405012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.427016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.437350] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.444012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.450013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.479016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.487468] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.497012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.529013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.539364] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.546015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.556015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.574015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.585000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.596014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.608014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.638014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.650195] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.652529] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.655456] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.658395] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.660222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.666031] iommu: Default domain type: Passthrough [ 0.668551] SCSI subsystem initialized [ 0.670151] ACPI: bus type USB registered [ 0.672208] usbcore: registered new interface driver usbfs [ 0.674069] usbcore: registered new interface driver hub [ 0.676089] usbcore: registered new device driver usb [ 0.678177] pps_core: LinuxPPS API ver. 1 registered [ 0.680011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.683117] PTP clock support registered [ 0.686083] EDAC MC: Ver: 3.0.0 [ 0.687485] PCI: Using ACPI for IRQ routing [ 0.689916] NetLabel: Initializing [ 0.691011] NetLabel: domain hash size = 128 [ 0.693009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.695074] NetLabel: unlabeled traffic allowed by default [ 0.697121] vgaarb: loaded [ 0.699264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.700009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.704480] clocksource: Switched to clocksource kvm-clock [ 0.816395] VFS: Disk quotas dquot_6.6.0 [ 0.818283] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821096] *** VALIDATE ramfs *** [ 0.822421] *** VALIDATE hugetlbfs *** [ 0.824095] pnp: PnP ACPI init [ 0.826473] pnp: PnP ACPI: found 6 devices [ 0.851523] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.856361] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.859595] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.862738] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.866284] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.870051] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.874341] NET: Registered protocol family 2 [ 0.876845] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.882074] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.886035] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.891694] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.895135] TCP: Hash tables configured (established 65536 bind 65536) [ 0.898202] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.901287] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904342] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.906638] NET: Registered protocol family 1 [ 0.910186] RPC: Registered named UNIX socket transport module. [ 0.912370] RPC: Registered udp transport module. [ 0.913808] RPC: Registered tcp transport module. [ 0.915304] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.917833] NET: Registered protocol family 44 [ 0.919611] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921808] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.924097] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.926630] PCI: CLS 0 bytes, default 64 [ 0.928528] Unpacking initramfs... [ 2.319850] debug: unmapping init [mem 0xffff8e517cc54000-0xffff8e517ffbffff] [ 2.323723] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.326083] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.329140] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.825941] Initialise system trusted keyrings [ 2.827766] Key type blacklist registered [ 2.829666] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.840166] zbud: loaded [ 2.844233] *** VALIDATE nfs *** [ 2.845317] *** VALIDATE nfs4 *** [ 2.847425] pstore: using deflate compression [ 2.853444] Platform Keyring initialized [ 2.953247] NET: Registered protocol family 38 [ 2.955134] Key type asymmetric registered [ 2.956238] Asymmetric key parser 'x509' registered [ 2.957872] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.961054] io scheduler mq-deadline registered [ 2.962601] io scheduler kyber registered [ 2.963924] io scheduler bfq registered [ 2.966115] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.969102] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.971728] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.974227] ACPI: Power Button [PWRF] [ 2.978968] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.985747] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.997829] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.004855] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.017819] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.046502] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.076538] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.082007] Non-volatile memory driver v1.3 [ 3.084179] Linux agpgart interface v0.103 [ 3.113656] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.116493] vda: detected capacity change from 0 to 76779520 [ 3.131790] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.134591] vdb: detected capacity change from 0 to 1073741824 [ 3.151849] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.155087] vdc: detected capacity change from 0 to 2621440000 [ 3.169285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.172711] vdd: detected capacity change from 0 to 2621440000 [ 3.188418] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.191980] vde: detected capacity change from 0 to 4294967296 [ 3.207745] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.210475] vdf: detected capacity change from 0 to 4294967296 [ 3.220960] libphy: Fixed MDIO Bus: probed [ 3.225761] usbcore: registered new interface driver usbserial_generic [ 3.228284] usbserial: USB Serial support registered for generic [ 3.230385] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.234349] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.236107] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.238401] mousedev: PS/2 mouse device common for all mice [ 3.240693] rtc_cmos 00:05: RTC can wake from S4 [ 3.243209] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.243611] rtc_cmos 00:05: registered as rtc0 [ 3.248344] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.251292] intel_pstate: CPU model not supported [ 3.253917] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.258126] hid: raw HID events driver (C) Jiri Kosina [ 3.258216] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.260434] usbcore: registered new interface driver usbhid [ 3.266220] usbhid: USB HID core driver [ 3.267580] drop_monitor: Initializing network drop monitor service [ 3.270286] Initializing XFRM netlink socket [ 3.272340] NET: Registered protocol family 10 [ 3.275419] Segment Routing with IPv6 [ 3.276929] NET: Registered protocol family 17 [ 3.279309] mpls_gso: MPLS GSO support [ 3.285782] RAS: Correctable Errors collector initialized. [ 3.287763] AVX version of gcm_enc/dec engaged. [ 3.289277] AES CTR mode by8 optimization enabled [ 3.364023] sched_clock: Marking stable (3364006189, 0)->(4334925325, -970919136) [ 3.367188] registered taskstats version 1 [ 3.369357] Loading compiled-in X.509 certificates [ 3.371660] zswap: loaded using pool lzo/zbud [ 3.395100] Key type big_key registered [ 3.407414] Key type encrypted registered [ 3.409220] ima: No TPM chip found, activating TPM-bypass! [ 3.411477] ima: Allocated hash algorithm: sha1 [ 3.413423] ima: No architecture policies found [ 3.414775] evm: Initialising EVM extended attributes: [ 3.416128] evm: security.selinux [ 3.416972] evm: security.ima [ 3.420635] evm: security.capability [ 3.421894] evm: HMAC attrs: 0x1 [ 3.424178] rtc_cmos 00:05: setting system clock to 2026-09-08 05:59:01 UTC (1788847141) [ 3.430573] debug: unmapping init [mem 0xffffffffbee03000-0xffffffffbeffffff] [ 3.433411] debug: unmapping init [mem 0xffffffffbdb82000-0xffffffffbde58fff] [ 3.442215] Write protecting the kernel read-only data: 28672k [ 3.445728] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.448878] debug: unmapping init [mem 0xffffffffbcb14000-0xffffffffbcbfffff] [ 3.482933] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.490536] systemd[1]: Detected virtualization kvm. [ 3.492872] systemd[1]: Detected architecture x86-64. [ 3.494769] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.518709] systemd[1]: No hostname configured. [ 3.520715] systemd[1]: Set hostname to . [ 3.522329] random: systemd: uninitialized urandom read (16 bytes read) [ 3.524565] systemd[1]: Initializing machine ID from random generator. [ 3.638870] random: systemd: uninitialized urandom read (16 bytes read) [ 3.643571] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.646681] random: systemd: uninitialized urandom read (16 bytes read) [ 3.649617] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.656149] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.221574] device-mapper: uevent: version 1.0.3 [ 4.223775] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.914502] virtio_net virtio0 ens2: renamed from eth0 [ 4.925062] random: fast init done [ 4.985509] scsi host0: ata_piix [ 5.028198] scsi host1: ata_piix [ 5.035455] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.040754] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.158079] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.928031] random: crng init done [ 9.929204] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.196811] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.308423] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.571712] SELinux: Disabled at runtime. [ 11.633498] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.642864] systemd[1]: Detected virtualization kvm. [ 11.644851] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.145789] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.148205] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.152193] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.155084] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.157693] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.169612] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.180354] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ 12.271301] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.599480] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.888939] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.939829] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.022365] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.032840] EDAC sbridge: Ver: 1.1.2 [ 14.907635] Key type dns_resolver registered [ 15.215488] NFS: Registering the id_resolver key type [ 15.218085] Key type id_resolver registered [ 15.219740] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg618-server login: [ 40.580752] libcfs: loading out-of-tree module taints kernel. [ 40.603369] Key type ._llcrypt registered [ 40.604934] Key type .llcrypt registered [ 40.651308] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 56.412370] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 57.845840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.863899] alg: No test for adler32 (adler32-zlib) [ 59.164699] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 60.129784] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 61.855201] Key type lgssc registered [ 63.491338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.193839] hrtimer: interrupt took 1909133 ns [ 78.772767] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 112.864358] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 123.390486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 123.420699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 124.716096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 124.805606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.910792] Lustre: lustre-MDT0000: new disk, initializing [ 124.973352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 125.007345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.042420] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.451725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.584818] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 144.658742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 144.664111] Lustre: Skipped 1 previous similar message [ 144.794333] Lustre: lustre-MDT0001: new disk, initializing [ 144.837465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.864586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.875817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.678865] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.373762] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 163.230165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.456910] Lustre: lustre-OST0000: new disk, initializing [ 163.461780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 163.468816] Lustre: 8461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.540731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.656746] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.063392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 172.069564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 172.129229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.584132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.678739] Lustre: lustre-OST0001: new disk, initializing [ 179.682049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 179.686767] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.742206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.481339] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.439331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 187.454715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 187.485078] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 197.667676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 205.060533] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 210.179963] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing check_logdir /tmp/testlogs/ [ 214.828291] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing yml_node [ 219.265866] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 221.815814] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 224.526590] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 225.913465] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 02:02:42 EDT 2026 [ 241.394536] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 242.981114] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 259.048367] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.048656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.076129] Lustre: Skipped 2 previous similar messages [ 259.080958] Lustre: Skipped 1 previous similar message [ 264.160193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 264.170216] Lustre: Skipped 5 previous similar messages [ 269.280830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.283614] Lustre: Skipped 3 previous similar messages [ 272.351126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 272.476836] Lustre: server umount lustre-MDT0000 complete [ 276.178789] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847414 with bad export cookie 12240503074212271025 [ 276.181565] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.191798] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 279.520209] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.522513] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.540389] Lustre: Skipped 1 previous similar message [ 279.543905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.587298] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 281.568025] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.582741] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 282.947255] Lustre: server umount lustre-MDT0001 complete [ 286.514405] Lustre: server umount lustre-OST0000 complete [ 290.213416] Lustre: server umount lustre-OST0001 complete [ 297.549542] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 307.616222] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 345.995181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.298647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.347788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.524947] Lustre: lustre-MDT0000: new disk, initializing [ 346.622240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.650765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.319950] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 361.984917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.135862] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 362.232590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 362.236523] Lustre: Skipped 1 previous similar message [ 362.369428] Lustre: lustre-MDT0001: new disk, initializing [ 362.474125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 362.507839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 362.521480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 367.126893] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 377.177441] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 382.876441] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 389.334417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 389.688367] Lustre: lustre-OST0000: new disk, initializing [ 389.692289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 389.703798] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 389.838243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.353961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 391.358594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 391.484365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 395.913625] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 405.182056] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 409.811891] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.095425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 415.424443] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 415.756220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 422.369425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 422.379628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.386546] LustreError: Skipped 1 previous similar message [ 422.388307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 422.406774] Lustre: Skipped 2 previous similar messages [ 422.426736] Lustre: Skipped 4 previous similar messages [ 424.112072] Lustre: server umount lustre-OST0000 complete [ 429.025223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 429.033062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.612134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 432.617758] Lustre: Skipped 2 previous similar messages [ 442.335092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.481660] Lustre: server umount lustre-MDT0000 complete [ 442.864630] LustreError: 15092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.883842] LustreError: 15092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 447.382542] LustreError: 17056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847585 with bad export cookie 12240503074212272558 [ 447.392549] LustreError: 17056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 447.394392] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 447.808897] Lustre: server umount lustre-MDT0001 complete [ 454.907541] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:06:31 (1788847591) [ 472.125570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.533556] LustreError: 19933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.608476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.638863] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 477.670443] LustreError: 19934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.941488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.012352] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 498.488663] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 504.097719] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 506.684912] Lustre: Failing over lustre-MDT0000 [ 506.849729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.860922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.871031] Lustre: Skipped 1 previous similar message [ 506.875554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 506.882577] Lustre: Skipped 2 previous similar messages [ 507.014072] Lustre: server umount lustre-MDT0000 complete [ 507.875919] LustreError: 19933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.909212] LustreError: 19933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 523.241169] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847645/real 1788847645] req@ffff8e51cfe89c00 x1875742239645696/t0(0) o400->MGC192.168.206.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788847661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 523.303790] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 525.755707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.355587] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.383215] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 533.477403] LustreError: 3639:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e51c3108a80 x1875742239648512/t0(0) o250->MGC192.168.206.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 538.020565] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 539.105138] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 539.115631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 539.136233] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 547.089407] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 553.449416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.458105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.466405] Lustre: Skipped 2 previous similar messages [ 553.477334] Lustre: Skipped 1 previous similar message [ 565.215134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 565.403953] Lustre: server umount lustre-MDT0000 complete [ 566.752490] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.784515] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 568.635871] LustreError: 19915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847706 with bad export cookie 12240503074212273972 [ 568.643839] LustreError: 19915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 568.644166] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.963476] Lustre: server umount lustre-MDT0001 complete [ 577.678565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.481411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.495090] Lustre: Skipped 2 previous similar messages [ 583.814651] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 592.294829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.968852] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 603.976938] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 609.367904] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 611.309671] Lustre: Failing over lustre-MDT0000 [ 611.536784] Lustre: server umount lustre-MDT0000 complete [ 613.345053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.729090] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847751/real 1788847751] req@ffff8e51fbdc4a80 x1875742239675136/t0(0) o400->MGC192.168.206.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788847767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 629.760736] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.489352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.848394] LustreError: 24513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.860885] LustreError: 24513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 639.383601] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 644.294961] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 644.580085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 644.590707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 644.596883] Lustre: Skipped 1 previous similar message [ 644.618842] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 654.770911] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 659.941213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.943821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 659.964372] Lustre: Skipped 1 previous similar message [ 659.979413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 660.007494] Lustre: Skipped 5 previous similar messages [ 673.247136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 673.522312] Lustre: server umount lustre-MDT0000 complete [ 677.377983] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847815 with bad export cookie 12240503074212274910 [ 677.384213] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.392311] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 677.580360] Lustre: server umount lustre-MDT0001 complete [ 688.690204] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:10:24 (1788847824) [ 705.149232] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 714.613898] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 758.372629] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 767.869142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.170556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 768.196910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 768.280689] Lustre: lustre-MDT0000: new disk, initializing [ 768.347260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.351323] Lustre: Skipped 2 previous similar messages [ 768.368626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 773.202648] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 784.117351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.289425] Lustre: 29652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 784.698544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 789.886439] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 801.123812] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 807.885635] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 814.235152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.836927] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 815.925365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 815.982370] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 816.003710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 816.015340] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 822.026564] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 832.023980] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 837.918932] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 862.177082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 862.183232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.189358] LustreError: Skipped 1 previous similar message [ 862.190079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 862.190083] Lustre: Skipped 6 previous similar messages [ 862.229225] Lustre: Skipped 2 previous similar messages [ 864.970694] Lustre: server umount lustre-OST0000 complete [ 871.904261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 875.783186] Lustre: server umount lustre-MDT0000 complete [ 877.540367] LustreError: 29663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.555990] LustreError: 29663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 880.101482] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848018 with bad export cookie 12240503074212275316 [ 880.104340] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.115160] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 880.679249] Lustre: server umount lustre-MDT0001 complete [ 892.673446] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 896.379207] Key type lgssc unregistered [ 896.817580] LNet: 33597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.840288] LNetError: 33597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.863882] LNet: Removed LNI 192.168.206.118@tcp [ 897.760641] Key type .llcrypt unregistered [ 897.763459] Key type ._llcrypt unregistered [ 924.437109] Key type ._llcrypt registered [ 924.439635] Key type .llcrypt registered [ 924.554497] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 925.561118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.575618] alg: No test for adler32 (adler32-zlib) [ 926.795519] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 927.075274] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 928.735210] Key type lgssc registered [ 930.424901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.274685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.299982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.054407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.773376] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 950.246720] LustreError: 35009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.273667] LustreError: 35009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 953.250852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.645361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.184263] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 970.414902] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 978.473924] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 986.622818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.125788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.911176] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 998.397621] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1004.016695] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1010.745959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1029.074483] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:16:05 (1788848165) [ 1031.239191] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1033.307307] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1040.282150] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:16:17 (1788848177) [ 1050.278466] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:16:26 (1788848186) [ 1051.396836] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1061.731884] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1061.743992] Lustre: Skipped 2 previous similar messages [ 1063.991813] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:16:40 (1788848200) [ 1066.368807] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1066.373848] Lustre: Skipped 2 previous similar messages [ 1074.782700] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1074.787066] Lustre: Skipped 4 previous similar messages [ 1091.976745] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:17:07 (1788848227) [ 1095.294481] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1100.296421] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1110.980876] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:17:27 (1788848247) [ 1112.833568] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1112.836863] Lustre: Skipped 6 previous similar messages [ 1126.336408] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:17:42 (1788848262) [ 1143.780240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.786578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.787600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1143.791989] Lustre: Skipped 1 previous similar message [ 1143.825160] Lustre: Skipped 1 previous similar message [ 1148.899523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.911135] Lustre: Skipped 1 previous similar message [ 1149.695932] Lustre: server umount lustre-OST0000 complete [ 1154.018535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.031778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.039621] Lustre: Skipped 1 previous similar message [ 1159.136834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.139586] Lustre: Skipped 1 previous similar message [ 1164.257084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.269088] Lustre: Skipped 1 previous similar message [ 1168.351807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1168.594570] Lustre: server umount lustre-MDT0000 complete [ 1169.376200] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.390647] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1172.247180] LustreError: 35742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848310 with bad export cookie 10065768710037717340 [ 1172.252956] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1172.265931] LustreError: 35742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1172.774833] Lustre: server umount lustre-MDT0001 complete [ 1183.737580] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1186.143402] Key type lgssc unregistered [ 1186.405529] LNet: 42886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.410471] LNetError: 42886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.425650] LNet: Removed LNI 192.168.206.118@tcp [ 1187.386312] Key type .llcrypt unregistered [ 1187.393423] Key type ._llcrypt unregistered [ 1205.934143] Key type ._llcrypt registered [ 1205.942188] Key type .llcrypt registered [ 1206.086293] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 1222.318558] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1223.493052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.619296] alg: No test for adler32 (adler32-zlib) [ 1224.742453] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1224.956453] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1226.696902] Key type lgssc registered [ 1227.924097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.356151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.378367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.639827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1265.673114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1265.772916] Lustre: lustre-MDT0000: new disk, initializing [ 1265.871605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.888583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1270.466993] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1280.698691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.808413] Lustre: 46039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1280.862379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1280.866775] Lustre: Skipped 1 previous similar message [ 1280.942712] Lustre: lustre-MDT0001: new disk, initializing [ 1281.042505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.102888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1281.115788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1285.618815] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1295.195504] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1300.810651] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1307.602626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1307.856358] Lustre: lustre-OST0000: new disk, initializing [ 1307.859905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1307.869299] Lustre: 48011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1307.944780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.090287] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1315.370846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1315.383857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1315.515072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1323.517834] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1327.675856] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1327.900978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1332.609817] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1332.791952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1335.776546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1335.782273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.802220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.302404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.305473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.308815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.330321] Lustre: Skipped 1 previous similar message [ 1340.839751] Lustre: server umount lustre-OST0000 complete [ 1346.535108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.548706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.559639] Lustre: Skipped 1 previous similar message [ 1351.648916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.654117] Lustre: Skipped 1 previous similar message [ 1356.772539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.781796] Lustre: Skipped 1 previous similar message [ 1358.303158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1358.474626] Lustre: server umount lustre-MDT0000 complete [ 1362.096054] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848500 with bad export cookie 3515319043088358981 [ 1362.097905] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.103915] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1362.450607] Lustre: server umount lustre-MDT0001 complete [ 1373.382733] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:21:49 (1788848509) [ 1380.987093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.573758] LustreError: 50471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.609392] LustreError: 50471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1381.664724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.986533] LustreError: 50472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.027856] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1391.077048] LustreError: 50471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.809612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.187693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.056350] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1414.436854] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1423.041735] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1431.231046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1431.673956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.698100] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1445.794884] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.695900] LustreError: 53002:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1456.032127] Lustre: Modifying parameter general.timeout=1 in log params [ 1457.212903] Lustre: Modifying parameter general.timeout=2 in log params [ 1460.106901] Lustre: Modifying parameter general.timeout=5 in log params [ 1460.112907] Lustre: Skipped 2 previous similar messages [ 1464.449739] Lustre: Modifying parameter general.timeout=9 in log params [ 1464.455647] Lustre: Skipped 3 previous similar messages [ 1473.156721] Lustre: Modifying parameter general.timeout=18 in log params [ 1473.159775] Lustre: Skipped 8 previous similar messages [ 1490.051176] Lustre: Modifying parameter general.timeout=37 in log params [ 1490.056089] Lustre: Skipped 18 previous similar messages [ 1522.949061] Lustre: Modifying parameter general.timeout=74 in log params [ 1522.959493] Lustre: Skipped 36 previous similar messages [ 1584.795673] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:25:21 (1788848721) [ 1587.767868] Lustre: 59951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1588.732618] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1588.750317] Lustre: Skipped 58 previous similar messages [ 1589.871313] Lustre: 60055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1591.919577] Lustre: 60153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1597.549891] Lustre: 60461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1597.560105] Lustre: 60461:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1625.463502] Lustre: 61821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1625.505296] LustreError: 61821:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1630.113475] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:26:06 (1788848766) [ 1645.537252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.550318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.560415] Lustre: Skipped 3 previous similar messages [ 1645.584869] Lustre: Skipped 2 previous similar messages [ 1649.549563] Lustre: server umount lustre-MDT0000 complete [ 1650.657394] LustreError: 51389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.689115] LustreError: 51389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1653.324243] LustreError: 50453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848791 with bad export cookie 3515319043088359863 [ 1653.328283] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1653.335472] LustreError: 50453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1653.899427] Lustre: server umount lustre-MDT0001 complete [ 1668.951263] Lustre: server umount lustre-OST0000 complete [ 1697.451508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.753385] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1698.123579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.553844] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1710.327725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.386385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1710.404874] Lustre: 64354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1715.346842] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1722.533559] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1728.807428] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.546360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.714116] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1735.911247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.917892] Lustre: Skipped 1 previous similar message [ 1741.043336] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1748.506803] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1750.387309] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1750.393647] Lustre: Skipped 20 previous similar messages [ 1750.432675] Lustre: 66920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1757.157118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.172561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.187618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.148883] Lustre: server umount lustre-OST0000 complete [ 1766.888990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.900445] LustreError: Skipped 1 previous similar message [ 1766.905824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.917222] Lustre: Skipped 1 previous similar message [ 1781.215139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1781.394658] Lustre: server umount lustre-MDT0000 complete [ 1783.265654] LustreError: 64367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1783.286435] LustreError: 64367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1785.017115] LustreError: 66225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848923 with bad export cookie 3515319043088362978 [ 1785.018132] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.036678] LustreError: 66225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1785.451728] Lustre: server umount lustre-MDT0001 complete [ 1796.393594] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1799.300290] Key type lgssc unregistered [ 1799.618904] LNet: 68130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.630212] LNetError: 68130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.643678] LNet: Removed LNI 192.168.206.118@tcp [ 1800.597665] Key type .llcrypt unregistered [ 1800.601966] Key type ._llcrypt unregistered [ 1827.714457] Key type ._llcrypt registered [ 1827.717204] Key type .llcrypt registered [ 1827.805513] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1828.879839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.913211] alg: No test for adler32 (adler32-zlib) [ 1829.987553] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1830.258307] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1831.975289] Key type lgssc registered [ 1833.126215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.988252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.004827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.553120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.483369] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1852.907587] LustreError: 69543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1852.925752] LustreError: 69543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1854.472271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.162215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.722672] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1867.464544] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1873.590194] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.087621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.798787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.733955] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1896.880289] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.608783] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1919.970971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.989510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.993104] Lustre: Skipped 1 previous similar message [ 1920.015394] Lustre: Skipped 2 previous similar messages [ 1922.752518] Lustre: server umount lustre-MDT0000 complete [ 1925.102627] LustreError: 69543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.146040] LustreError: 69543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1926.411899] LustreError: 72231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849064 with bad export cookie 9572513947450045703 [ 1926.415725] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.428426] LustreError: 72231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1926.768304] Lustre: server umount lustre-MDT0001 complete [ 1940.853874] Lustre: server umount lustre-OST0000 complete [ 1965.573073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.870565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1965.906460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1966.207684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.705570] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1979.147892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.263961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1979.303410] Lustre: 74418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1979.809371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1984.054959] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1991.248493] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1997.624529] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2003.689649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.823115] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2004.199470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.644679] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2017.195236] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2018.978970] Lustre: 76992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2020.783348] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2029.540038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.544719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2029.548469] Lustre: Skipped 2 previous similar messages [ 2029.550051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2029.579104] Lustre: Skipped 1 previous similar message [ 2032.799730] Lustre: server umount lustre-OST0000 complete [ 2039.783724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.789625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.798240] Lustre: Skipped 1 previous similar message [ 2039.813808] Lustre: Skipped 1 previous similar message [ 2042.757538] Lustre: server umount lustre-MDT0000 complete [ 2044.896278] LustreError: 74428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.904969] LustreError: 74428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2046.676939] LustreError: 76285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849184 with bad export cookie 9572513947450046543 [ 2046.680239] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.684686] LustreError: 76285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2047.013053] Lustre: server umount lustre-MDT0001 complete [ 2060.308328] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2063.376860] Key type lgssc unregistered [ 2063.637189] LNet: 78201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.649145] LNetError: 78201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.671609] LNet: Removed LNI 192.168.206.118@tcp [ 2064.646410] Key type .llcrypt unregistered [ 2064.651057] Key type ._llcrypt unregistered [ 2092.179627] Key type ._llcrypt registered [ 2092.182129] Key type .llcrypt registered [ 2092.306221] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2093.631180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2093.653302] alg: No test for adler32 (adler32-zlib) [ 2094.794710] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2095.078594] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2096.767272] Key type lgssc registered [ 2097.777787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2105.136190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2105.147608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.502693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.307375] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2116.067440] LustreError: 79613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.105478] LustreError: 79613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2119.369950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.799815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.699780] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2130.703145] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2136.457728] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2142.626443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.019471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.293512] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2155.338414] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2161.162670] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2161.826248] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2163.238392] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2163.242950] Lustre: Skipped 1 previous similar message [ 2165.312707] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2165.316578] Lustre: Skipped 2 previous similar messages [ 2169.726864] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2169.730973] Lustre: Skipped 5 previous similar messages [ 2177.941426] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2177.953366] Lustre: Skipped 10 previous similar messages [ 2194.154664] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2194.160644] Lustre: Skipped 21 previous similar messages [ 2226.531974] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.536554] Lustre: Skipped 43 previous similar messages [ 2240.465552] Lustre: 87707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2242.848916] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:36:19 (1788849379) [ 2244.390121] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2245.927678] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:36:22 (1788849382) [ 2248.678229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.684619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.697484] Lustre: Skipped 1 previous similar message [ 2248.710340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.795959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.810575] Lustre: Skipped 2 previous similar messages [ 2254.820170] Lustre: server umount lustre-OST0000 complete [ 2257.889887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.899731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.911820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.918806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.039889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.047031] Lustre: Skipped 3 previous similar messages [ 2269.159212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.165815] Lustre: Skipped 1 previous similar message [ 2272.223624] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2272.362759] Lustre: server umount lustre-MDT0000 complete [ 2274.273377] LustreError: 79612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.296886] LustreError: 79612:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2275.901199] LustreError: 82201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849413 with bad export cookie 5790676350466867760 [ 2275.905320] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2275.915284] LustreError: 82201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2276.162607] Lustre: server umount lustre-MDT0001 complete [ 2284.970266] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2287.261646] Key type lgssc unregistered [ 2287.520216] LNet: 88954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.531267] LNetError: 88954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.556648] LNet: Removed LNI 192.168.206.118@tcp [ 2288.360274] Key type .llcrypt unregistered [ 2288.365482] Key type ._llcrypt unregistered [ 2292.223830] Key type ._llcrypt registered [ 2292.226183] Key type .llcrypt registered [ 2292.302822] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2296.007675] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules [ 2296.888343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2296.916905] alg: No test for adler32 (adler32-zlib) [ 2297.962749] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2298.170582] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2299.823284] Key type lgssc registered [ 2300.837416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2301.814609] LustreError: 90151:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2316.688303] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2322.951408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.308065] LustreError: 90151:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2323.333877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.750509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.075087] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2334.187460] LustreError: 91175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.204584] LustreError: 91175:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2334.391464] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:37:51 (1788849471) [ 2337.846357] LustreError: 91207:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2337.853961] LustreError: 91207:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2337.859211] LustreError: 91207:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 2338.084134] Lustre: server umount lustre-MDT0000 complete [ 2349.167428] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2351.353855] Key type lgssc unregistered [ 2351.606453] LNet: 92698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.614680] LNetError: 92698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.642709] LNet: Removed LNI 192.168.206.118@tcp [ 2352.347533] Key type .llcrypt unregistered [ 2352.350160] Key type ._llcrypt unregistered [ 2366.626613] Key type ._llcrypt registered [ 2366.632200] Key type .llcrypt registered [ 2366.729954] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2367.462094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.495996] alg: No test for adler32 (adler32-zlib) [ 2368.523797] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2368.752321] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2370.423183] Key type lgssc registered [ 2371.275337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2377.450171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2377.464596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.874662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.561266] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2388.451906] LustreError: 93912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.474340] LustreError: 93912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2390.605472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.238393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2395.020932] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2401.047794] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.559444] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2411.094735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2411.558262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.558600] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2422.789074] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2468.401586] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:40:05 (1788849605) [ 2469.963922] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2471.463250] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:40:08 (1788849608) [ 2474.979432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.980069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.987058] Lustre: Skipped 2 previous similar messages [ 2474.993754] Lustre: Skipped 2 previous similar messages [ 2480.101532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.111886] Lustre: Skipped 2 previous similar messages [ 2481.091198] Lustre: server umount lustre-MDT0000 complete [ 2483.943102] LustreError: 93893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849622 with bad export cookie 2715466267574484649 [ 2483.945739] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.954389] LustreError: 93893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2484.285463] Lustre: server umount lustre-MDT0001 complete [ 2497.840746] Lustre: server umount lustre-OST0000 complete [ 2506.315838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.734289] LustreError: 98187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.859819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.536801] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2512.352711] LustreError: 98188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.450171] LustreError: 98187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.908645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.468844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2522.348604] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2528.574754] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2532.933023] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2542.533000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2542.689314] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2542.694372] LustreError: 98176:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2542.712279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2542.722548] LustreError: 100270:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2542.727198] LustreError: 100270:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2542.730388] LustreError: 100270:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2542.779894] Lustre: server umount lustre-OST0000 complete [ 2542.784482] LustreError: 100270:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2549.494361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2549.661120] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2549.667436] LustreError: 98177:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2549.687281] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2549.696029] LustreError: 100736:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2549.704845] LustreError: 100736:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2549.709059] LustreError: 100736:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2549.756361] Lustre: server umount lustre-OST0000 complete [ 2549.758590] LustreError: 100736:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2556.224545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2556.376361] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2556.379770] Lustre: Found index 0 for lustre-OST0000, updating log [ 2556.385383] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2556.433160] Lustre: lustre-OST0000: new disk, initializing [ 2556.438251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2556.448272] Lustre: 101267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2556.671180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.183278] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2563.862601] Lustre: Failing over lustre-OST0000 [ 2563.985740] Lustre: server umount lustre-OST0000 complete [ 2569.187188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.188699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.196721] Lustre: Skipped 1 previous similar message [ 2573.037578] Lustre: server umount lustre-MDT0000 complete [ 2574.304220] LustreError: 98188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.321240] LustreError: 98188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2575.466816] LustreError: 102025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849713 with bad export cookie 2715466267574487393 [ 2575.474495] LustreError: 102025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2575.474613] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.213290] Lustre: server umount lustre-MDT0001 complete [ 2591.874952] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:42:08 (1788849728) [ 2603.760506] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2611.345604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.753838] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.870512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.817609] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2621.127739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.751702] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2627.321040] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2631.926807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.522670] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2637.256353] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2637.671183] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2637.681155] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2637.696164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2642.920433] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2642.934766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2643.727579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.803479] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2643.881270] Lustre: lustre-OST0001: new disk, initializing [ 2643.886463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2643.893039] Lustre: 105937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2644.141558] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2644.151971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2644.153655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2644.162752] Lustre: Skipped 2 previous similar messages [ 2646.189670] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:01 [ 2646.199640] Lustre: lustre-OST0000: Recovery over after 0:09, of 1 clients 1 recovered and 0 were evicted. [ 2646.200152] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2646.202921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2649.147822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2649.157053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2649.273395] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2649.376961] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2655.501341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2663.293150] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2666.685078] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.18@tcp (stopping) [ 2666.695549] Lustre: Skipped 1 previous similar message [ 2669.027512] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.036697] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.785741] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.18@tcp (stopping) [ 2671.795303] Lustre: Skipped 4 previous similar messages [ 2672.352625] Lustre: server umount lustre-MDT0001 complete [ 2675.171208] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.181370] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2678.773474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.864480] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2678.872159] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2678.918175] Lustre: 103610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2680.243441] LustreError: 107518:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2680.250883] LustreError: 107518:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2680.261532] LustreError: 103612:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2680.323413] LustreError: 107523:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.118@tcp: cfg command failed: rc = -17 [ 2680.329449] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2680.329449] [ 2682.362633] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2684.904674] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2688.882363] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:43:46 (1788849826) [ 2700.944881] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2712.149959] Lustre: 109552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2728.234722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2730.895518] Lustre: 110686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2807.263816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.267772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.276106] Lustre: Skipped 2 previous similar messages [ 2807.280369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2809.220249] Lustre: server umount lustre-MDT0000 complete [ 2811.387621] LustreError: 103602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849949 with bad export cookie 2715466267574488324 [ 2811.391598] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.397485] LustreError: 103602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2811.559407] Lustre: server umount lustre-MDT0001 complete [ 2824.296843] Lustre: server umount lustre-OST0000 complete [ 2836.828225] Lustre: server umount lustre-OST0001 complete [ 2840.191883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2846.528682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2851.965195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2858.416838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2870.172319] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2876.372727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.387499] Lustre: lustre-MDT0000: reset Object Index mappings [ 2876.674114] LustreError: 114639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.686175] LustreError: 114639:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2876.741062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.748128] Lustre: Skipped 1 previous similar message [ 2879.398170] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2884.956266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.979683] Lustre: lustre-MDT0001: reset Object Index mappings [ 2888.570347] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2890.501790] Lustre: 115787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2894.630556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.171505] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 2898.702295] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2901.303093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2904.769195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.557224] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2910.195043] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2910.200403] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2915.171879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2917.624026] Lustre: 117661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2953.764184] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:48:10 (1788850090) [ 2981.857618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.863473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.865806] Lustre: Skipped 6 previous similar messages [ 2981.872482] Lustre: Skipped 6 previous similar messages [ 2987.383610] Lustre: server umount lustre-MDT0000 complete [ 2989.860989] LustreError: 114620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850127 with bad export cookie 2715466267574535154 [ 2989.867752] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2989.870194] LustreError: 114620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2992.100462] LustreError: 114635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.100912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.113117] LustreError: 114635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2992.118840] Lustre: Skipped 1 previous similar message [ 3004.383173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3004.592791] Lustre: server umount lustre-MDT0001 complete [ 3013.601935] Lustre: server umount lustre-OST0000 complete [ 3016.129166] LustreError: 118618:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x3c:0x0].0x0 (ffff8e51f0fa1f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3022.285853] Lustre: server umount lustre-OST0001 complete [ 3025.874775] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 3031.745455] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3050.530208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.628104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3050.641832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3050.681297] Lustre: lustre-MDT0000: new disk, initializing [ 3050.718507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.721512] Lustre: Skipped 3 previous similar messages [ 3050.728528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3052.482763] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3056.583140] Lustre: server umount lustre-MDT0000 complete [ 3062.369467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.538612] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3062.545708] Lustre: 121924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3062.553598] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3065.339148] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3068.425784] Lustre: server umount lustre-MDT0000 complete [ 3074.041790] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:50:11 (1788850211) [ 3075.168737] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3076.401712] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:50:13 (1788850213) [ 3087.576857] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3093.385627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.936654] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3100.594562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.664825] Lustre: 123973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3100.707154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3100.709476] Lustre: Skipped 1 previous similar message [ 3100.759351] Lustre: lustre-MDT0001: new disk, initializing [ 3100.850130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3100.856418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3103.450979] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3106.703417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3110.387529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.562677] Lustre: lustre-OST0000: new disk, initializing [ 3110.566670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3110.571395] Lustre: 125550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3112.520972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3112.526143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3112.580071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3114.106046] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3120.313242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.410344] Lustre: lustre-OST0001: new disk, initializing [ 3120.414753] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3120.419131] Lustre: 126418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3121.952329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3121.959898] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3122.042845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3123.766771] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3130.242636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3132.432148] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3372.345675] Lustre: 132809:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3372.349910] Lustre: 132809:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3372.353072] Lustre: 132809:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3372.355829] Lustre: 132809:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3372.359448] Lustre: 132809:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3372.363080] Lustre: 132809:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3500.380821] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:57:17 (1788850637) [ 3515.871975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3515.875116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.881314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.883811] Lustre: Skipped 12 previous similar messages [ 3518.533202] Lustre: server umount lustre-MDT0000 complete [ 3520.006684] LustreError: 132843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850658 with bad export cookie 2715466267574599246 [ 3520.011530] LustreError: 132843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.011607] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3521.504928] LustreError: 123979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.515390] LustreError: 123979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3534.303116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3534.435850] Lustre: server umount lustre-MDT0001 complete [ 3542.221682] Lustre: server umount lustre-OST0000 complete [ 3558.367111] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3558.459414] Lustre: server umount lustre-OST0001 complete [ 3561.414345] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 3566.163979] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3583.705630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.824966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3583.845247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3583.885151] Lustre: lustre-MDT0000: new disk, initializing [ 3583.920912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.923989] Lustre: Skipped 5 previous similar messages [ 3583.931868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3585.824335] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3591.278226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.332300] Lustre: 135830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3591.461838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3593.274054] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3597.851135] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3600.154168] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3603.040349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.184928] Lustre: 137800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3605.121069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3605.124776] Lustre: Skipped 1 previous similar message [ 3605.127154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3605.173789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3605.694856] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3609.986366] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3613.994365] Lustre: lustre-MDD0000: changelog on [ 3615.099474] Lustre: lustre-MDD0001: changelog on [ 4283.383721] Lustre: lustre-MDD0001: changelog off [ 4284.098257] Lustre: 141457:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4284.457664] Lustre: lustre-MDD0000: changelog off [ 4291.552569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4291.553141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.555250] LustreError: Skipped 1 previous similar message [ 4291.561393] Lustre: Skipped 5 previous similar messages [ 4291.563748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4291.566994] Lustre: Skipped 11 previous similar messages [ 4292.637598] Lustre: server umount lustre-OST0000 complete [ 4296.674411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.682037] Lustre: Skipped 1 previous similar message [ 4300.332619] Lustre: server umount lustre-MDT0000 complete [ 4301.792505] LustreError: 136893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.799297] LustreError: 136893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4302.135656] LustreError: 135825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851440 with bad export cookie 2715466267575368462 [ 4302.138706] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4302.142285] LustreError: 135825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4302.297848] Lustre: server umount lustre-MDT0001 complete [ 4306.697454] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 4308.061755] Key type lgssc unregistered [ 4308.213662] LNet: 146612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4308.216954] LNetError: 146612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4308.226363] LNet: Removed LNI 192.168.206.118@tcp [ 4308.658229] Key type .llcrypt unregistered [ 4308.660130] Key type ._llcrypt unregistered [ 4316.370901] Key type ._llcrypt registered [ 4316.373542] Key type .llcrypt registered [ 4316.419446] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 4323.436167] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4323.905504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4323.916388] alg: No test for adler32 (adler32-zlib) [ 4324.794449] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 4324.904161] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 4326.495218] Key type lgssc registered [ 4326.924712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4340.625342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4340.636865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.734360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4341.746955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4341.782342] Lustre: lustre-MDT0000: new disk, initializing [ 4341.808584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.816955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4343.741779] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4348.676163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.723284] Lustre: 149781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4348.752454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4348.754572] Lustre: Skipped 1 previous similar message [ 4348.791028] Lustre: lustre-MDT0001: new disk, initializing [ 4348.823995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4348.838925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4348.843029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4350.199615] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4354.034741] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4356.071245] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4358.475240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.584614] Lustre: lustre-OST0000: new disk, initializing [ 4358.587100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4358.592204] Lustre: 151749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4358.631961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.650276] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4364.453309] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4365.307690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4365.311688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4365.350429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4372.321624] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_module obdecho/obdecho [ 4372.974350] LustreError: 152753:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4372.977410] LustreError: 152753:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4373.642975] LustreError: 152849:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4379.616146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4379.618929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.624762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.640334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.641838] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.198406] Lustre: server umount lustre-MDT0000 complete [ 4384.571727] LustreError: 149774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851522 with bad export cookie 8850517628412092368 [ 4384.574958] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4384.576940] LustreError: 149774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4384.689923] Lustre: server umount lustre-MDT0001 complete [ 4396.396848] Lustre: server umount lustre-OST0000 complete [ 4404.685871] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 4409.859429] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4425.119906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.215899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4425.232393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4425.271512] Lustre: lustre-MDT0000: new disk, initializing [ 4425.314831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4425.322101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4427.701558] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4433.227209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.290779] Lustre: 156556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4433.326422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4433.329267] Lustre: Skipped 1 previous similar message [ 4433.370287] Lustre: lustre-MDT0001: new disk, initializing [ 4433.416901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4433.434059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4433.438238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4435.230119] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4439.867430] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4442.565340] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4445.551935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.686812] Lustre: lustre-OST0000: new disk, initializing [ 4445.689611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4445.693698] Lustre: 158525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4445.740199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.311482] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4452.370351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4452.375895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4452.419470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4452.633129] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4454.401466] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4454.508203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4456.151763] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4456.243781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4457.441791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.442249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.445360] LustreError: Skipped 1 previous similar message [ 4457.446884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4457.449311] Lustre: Skipped 2 previous similar messages [ 4457.451583] Lustre: Skipped 2 previous similar messages [ 4459.085332] Lustre: server umount lustre-OST0000 complete [ 4462.560574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.561310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.567399] Lustre: Skipped 1 previous similar message [ 4462.569606] Lustre: Skipped 1 previous similar message [ 4466.475270] Lustre: server umount lustre-MDT0000 complete [ 4468.078461] LustreError: 156547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851606 with bad export cookie 8850517628412093320 [ 4468.082808] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4468.087906] LustreError: 156547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4468.254654] Lustre: server umount lustre-MDT0001 complete [ 4470.116782] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:13:27 (1788851607) [ 4470.911586] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4471.792216] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:13:29 (1788851609) [ 4475.150801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.344677] LustreError: 160930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.353349] LustreError: 160930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4475.378552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.229994] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4480.482392] LustreError: 160931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.122493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.197765] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4485.984620] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4487.881503] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4490.404392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.901810] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4495.564193] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4503.408291] LustreError: 163506:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x784aa6ba (000000001bee0485) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4506.081860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.083068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4506.087829] Lustre: Skipped 2 previous similar messages [ 4506.089816] Lustre: Skipped 2 previous similar messages [ 4509.556939] Lustre: server umount lustre-MDT0000 complete [ 4511.091926] LustreError: 160910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851649 with bad export cookie 8850517628412094188 [ 4511.095682] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.096914] LustreError: 160910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4511.200513] LustreError: 163417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.319826] LustreError: 162766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.320332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4516.327851] LustreError: 162766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4516.330013] Lustre: Skipped 4 previous similar messages [ 4517.334649] Lustre: server umount lustre-MDT0001 complete [ 4519.831438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4524.697749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.795175] LustreError: 164768:0:(llog.c:1655:llog_backup()) MGC192.168.206.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4524.800089] Lustre: 164768:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4541.092894] LustreError: 164773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.100161] LustreError: 164773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4551.137139] LustreError: 164774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.144074] LustreError: 164774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4567.716950] LustreError: 164792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.724819] LustreError: 164792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 4585.567332] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4585.570962] Lustre: 164768:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4602.336299] LustreError: 164775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.350564] LustreError: 164775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 4626.414942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4626.421870] Lustre: Skipped 2 previous similar messages [ 4628.688519] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4631.547781] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4634.214520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.096483] LustreError: 148007:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ea4b0163 x1875746710722816/t0(0) o250->MGC192.168.206.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4656.111476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4657.995777] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4660.947075] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.701850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4661.736574] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4664.135933] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4668.932239] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4677.091368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4677.092522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.094941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4677.094952] Lustre: Skipped 1 previous similar message [ 4677.105992] Lustre: Skipped 3 previous similar messages [ 4680.286050] Lustre: server umount lustre-MDT0000 complete [ 4681.883418] LustreError: 160911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851819 with bad export cookie 8850517628412100663 [ 4681.886434] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.891239] LustreError: 160911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4682.020608] Lustre: server umount lustre-MDT0001 complete [ 4693.860394] Lustre: server umount lustre-OST0000 complete [ 4701.536778] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 4705.971784] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4720.864911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.971675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4720.987982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4721.028865] Lustre: lustre-MDT0000: new disk, initializing [ 4721.060550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.063705] Lustre: Skipped 1 previous similar message [ 4721.070745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4722.725171] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4727.545240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.592641] Lustre: 169847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4727.716740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4729.275166] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4734.096097] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4736.614281] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4739.574303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.701770] Lustre: 171815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4742.176565] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4746.731993] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4748.296329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4748.348916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4748.587417] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4748.677546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4750.418394] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4750.518803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4753.378715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4753.379420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.380296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4753.380301] Lustre: Skipped 2 previous similar messages [ 4753.382200] LustreError: Skipped 1 previous similar message [ 4753.386466] Lustre: Skipped 1 previous similar message [ 4757.456032] Lustre: server umount lustre-OST0000 complete [ 4763.617613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4773.343091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4773.402321] Lustre: server umount lustre-MDT0000 complete [ 4773.855922] LustreError: 170659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.862068] LustreError: 170659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 4774.633303] LustreError: 171814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851912 with bad export cookie 8850517628412117463 [ 4774.637265] LustreError: 171814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4774.637886] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.752319] Lustre: server umount lustre-MDT0001 complete [ 4777.892827] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:18:35 (1788851915) [ 4780.242320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.173361] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4787.708078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.473662] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4795.235268] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4799.257056] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4803.496089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4806.877080] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4811.017691] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.212990] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:19:17 (1788851957) [ 4823.525358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4823.527281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.531402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4823.536492] Lustre: Skipped 3 previous similar messages [ 4823.541534] Lustre: Skipped 5 previous similar messages [ 4827.931186] Lustre: server umount lustre-OST0000 complete [ 4835.500548] Lustre: server umount lustre-MDT0000 complete [ 4837.215924] LustreError: 174207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851975 with bad export cookie 8850517628412118331 [ 4837.219798] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.221526] LustreError: 174207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4837.381771] Lustre: server umount lustre-MDT0001 complete [ 4842.981214] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 4844.679538] Key type lgssc unregistered [ 4844.920134] LNet: 177950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4844.929187] LNetError: 177950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4844.942382] LNet: Removed LNI 192.168.206.118@tcp [ 4845.953868] Key type .llcrypt unregistered [ 4845.956606] Key type ._llcrypt unregistered [ 4856.210515] Key type ._llcrypt registered [ 4856.212115] Key type .llcrypt registered [ 4856.263727] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4856.731123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4856.790062] alg: No test for adler32 (adler32-zlib) [ 4857.698901] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 4857.824730] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 4859.423249] Key type lgssc registered [ 4860.116878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4864.372842] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4864.396403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4879.903494] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4885.023376] LustreError: 179167:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 4890.143573] LustreError: 179167:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4890.149220] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4890.152189] LustreError: MGC192.168.206.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4890.158354] LustreError: 179167:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4890.162574] LustreError: 179167:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 4890.166588] LustreError: 179167:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 4890.197793] Lustre: server umount lustre-OST0000 complete [ 4890.199591] LustreError: 179167:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4893.817971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.000977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.809572] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4900.326343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.477086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4902.219268] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4905.792368] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4908.718114] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4911.786372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4911.882251] LustreError: 181479:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4911.887300] Lustre: 181479:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4911.942033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.354422] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4917.331932] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4918.216978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.218119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4918.222236] Lustre: Skipped 2 previous similar messages [ 4918.224565] Lustre: Skipped 1 previous similar message [ 4923.360908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4923.364793] Lustre: Skipped 3 previous similar messages [ 4924.414409] Lustre: server umount lustre-MDT0000 complete [ 4925.409027] LustreError: 182051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.420607] LustreError: 182051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4926.224441] LustreError: 179647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852064 with bad export cookie 11705799477074838626 [ 4926.229319] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.233164] LustreError: 179647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4927.455642] LustreError: 180395:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000040a6fcc9 x1875747269529600/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4927.528171] Lustre: server umount lustre-MDT0001 complete [ 4939.430706] Lustre: server umount lustre-OST0000 complete [ 4943.592785] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 4947.575876] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4962.129383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.235802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4962.257648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4962.297680] Lustre: lustre-MDT0000: new disk, initializing [ 4962.330970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4962.340963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4963.890092] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4968.903952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.960909] Lustre: 185000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4968.992759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4968.995574] Lustre: Skipped 1 previous similar message [ 4969.035382] Lustre: lustre-MDT0001: new disk, initializing [ 4969.077887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4969.097114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4969.101983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4970.966859] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4975.673131] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4977.899488] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4980.353232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.462873] Lustre: lustre-OST0000: new disk, initializing [ 4980.464764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4980.468106] Lustre: 186967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4980.502188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.607442] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4986.725495] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4988.401943] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4989.434752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4989.438534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4989.482677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4990.543858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4992.318489] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4992.454311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4994.530169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4994.531749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.534385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4994.541527] Lustre: Skipped 1 previous similar message [ 4995.422739] Lustre: server umount lustre-OST0000 complete [ 4999.648608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.650603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4999.656990] Lustre: Skipped 1 previous similar message [ 4999.658822] Lustre: Skipped 1 previous similar message [ 5003.342775] Lustre: server umount lustre-MDT0000 complete [ 5004.769817] LustreError: 185010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.784282] LustreError: 185010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5005.763778] LustreError: 184991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852143 with bad export cookie 11705799477074839319 [ 5005.769729] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.771436] LustreError: 184991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5006.030574] Lustre: server umount lustre-MDT0001 complete [ 5011.284136] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:22:28 (1788852148) [ 5022.127064] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5023.854442] Key type lgssc unregistered [ 5024.042475] LNet: 189922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5024.046803] LNetError: 189922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5024.074675] LNet: Removed LNI 192.168.206.118@tcp [ 5024.599245] Key type .llcrypt unregistered [ 5024.601071] Key type ._llcrypt unregistered [ 5034.076668] Key type ._llcrypt registered [ 5034.078141] Key type .llcrypt registered [ 5034.147121] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5034.623176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5034.667695] alg: No test for adler32 (adler32-zlib) [ 5035.556170] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 5035.672871] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5037.271261] Key type lgssc registered [ 5037.716588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5040.684152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5040.691821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.853238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5043.407254] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5046.624547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.811694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5048.408405] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5051.730045] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5054.552691] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5058.430498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5058.550887] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5058.558183] LustreError: 192975:0:(llog.c:1696:llog_backup()) MGC192.168.206.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5058.563127] Lustre: 192975:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5058.640122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.760950] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5065.947354] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5068.769691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.775138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5068.777309] Lustre: Skipped 2 previous similar messages [ 5068.784771] Lustre: Skipped 1 previous similar message [ 5071.905656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.911294] Lustre: Skipped 3 previous similar messages [ 5073.550545] Lustre: server umount lustre-MDT0000 complete [ 5075.766505] LustreError: 191094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852213 with bad export cookie 9383007466056988550 [ 5075.773217] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.785334] LustreError: 191094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5075.955602] LustreError: 191842:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001c5103d5 x1875747456175616/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5076.048450] Lustre: server umount lustre-MDT0001 complete [ 5088.948518] Lustre: server umount lustre-OST0000 complete [ 5093.506624] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 5098.002955] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5113.962038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.095952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5114.118444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5114.167462] Lustre: lustre-MDT0000: new disk, initializing [ 5114.213550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.231122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5116.294591] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5122.390376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.484797] Lustre: 196493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5122.533884] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5122.535923] Lustre: Skipped 1 previous similar message [ 5122.537702] LustreError: 197274:0:(llog.c:1696:llog_backup()) MGC192.168.206.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5122.541163] LustreError: 197274:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5122.544039] Lustre: 197274:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5122.555896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5122.558729] Lustre: Skipped 1 previous similar message [ 5122.611749] Lustre: lustre-MDT0001: new disk, initializing [ 5122.669826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.690456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5122.696640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5124.916870] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5131.533650] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5134.622589] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5138.020293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5138.141561] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5138.143550] Lustre: Skipped 1 previous similar message [ 5138.145607] LustreError: 198433:0:(llog.c:1696:llog_backup()) MGC192.168.206.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5138.149376] LustreError: 198433:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5138.152337] Lustre: 198433:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5138.156923] Lustre: 198433:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5138.170351] Lustre: lustre-OST0000: new disk, initializing [ 5138.172702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5138.176291] Lustre: 198458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5138.224063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.163759] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5146.446600] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5147.678158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5147.687087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5147.741608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5148.600516] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5148.714659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5150.708996] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5150.817571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5152.737812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5152.740762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.747306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5152.750490] Lustre: Skipped 1 previous similar message [ 5157.876874] Lustre: server umount lustre-OST0000 complete [ 5162.976990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5162.978106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5162.983813] Lustre: Skipped 2 previous similar messages [ 5168.097450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5168.100862] Lustre: Skipped 2 previous similar messages [ 5174.239201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5174.329297] Lustre: server umount lustre-MDT0000 complete [ 5176.153502] LustreError: 196485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852314 with bad export cookie 9383007466056989236 [ 5176.156351] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.164959] LustreError: 196485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5176.332738] Lustre: server umount lustre-MDT0001 complete [ 5180.759103] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:25:18 (1788852318) [ 5187.904794] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5192.379937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.572380] LustreError: 201581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.580873] LustreError: 201581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5192.598786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.451140] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5197.793662] LustreError: 201582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.231301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.389060] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5201.858954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5204.961178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5208.127968] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5212.388217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5212.474809] Lustre: lustre-OST0001: new disk, initializing [ 5212.477614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5212.482208] Lustre: 203885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5215.259371] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5215.582448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5215.586231] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5215.627551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5221.547402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5224.579807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5225.403121] Lustre: DEBUG MARKER: ADD OST3 [ 5227.413769] loop: module loaded [ 5227.434110] loop0: detected capacity change from 0 to 204800000 [ 5227.564798] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5227.571138] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5227.579815] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5230.398363] loop0: detected capacity change from 0 to 204800000 [ 5232.813674] Lustre: DEBUG MARKER: STOP OST3 [ 5232.899217] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5232.981867] Lustre: lustre-OST0003: new disk, initializing [ 5232.983964] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5232.987949] Lustre: 205675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5233.040649] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5233.043623] Lustre: Skipped 3 previous similar messages [ 5233.639385] Lustre: Failing over lustre-OST0003 [ 5233.678617] Lustre: server umount lustre-OST0003 complete [ 5234.244485] LustreError: 204416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.136523] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5239.469130] LustreError: 204416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.481304] LustreError: 204416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5241.530735] Lustre: DEBUG MARKER: START OST3 again [ 5244.599455] LustreError: 204283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.610605] LustreError: 204283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5245.956424] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5249.861389] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5251.583061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5251.595098] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5251.635074] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5265.797684] Lustre: Failing over lustre-OST0003 [ 5265.875799] Lustre: server umount lustre-OST0003 complete [ 5266.911882] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.922937] LustreError: Skipped 1 previous similar message [ 5266.925774] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.926986] LustreError: 203065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.926993] LustreError: 203065:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5266.969070] Lustre: Skipped 1 previous similar message [ 5280.224473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5280.229184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.242461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.245113] Lustre: Skipped 2 previous similar messages [ 5286.314117] Lustre: server umount lustre-MDT0000 complete [ 5287.393932] LustreError: 201582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.409444] LustreError: 201582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5288.461742] LustreError: 204657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852426 with bad export cookie 9383007466056990111 [ 5288.465247] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.477087] LustreError: 204657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5292.515702] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.522035] Lustre: Skipped 4 previous similar messages [ 5294.672732] Lustre: server umount lustre-MDT0001 complete [ 5296.549391] Lustre: server umount lustre-OST0000 complete [ 5298.382844] Lustre: server umount lustre-OST0001 complete [ 5301.251513] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 5305.659233] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5323.039522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.135630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5323.152756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5323.191914] Lustre: lustre-MDT0000: new disk, initializing [ 5323.241764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5323.244948] Lustre: Skipped 1 previous similar message [ 5323.252522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5324.922583] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5329.424628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.469890] Lustre: 211185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5329.577845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5331.051077] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5335.149863] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5337.458681] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5339.878815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5339.984962] Lustre: lustre-OST0000: new disk, initializing [ 5339.986788] Lustre: Skipped 1 previous similar message [ 5339.989071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5339.991718] Lustre: Skipped 2 previous similar messages [ 5339.993994] Lustre: 213154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5341.690746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5341.693639] Lustre: Skipped 1 previous similar message [ 5341.695736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5341.731794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5342.162145] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5346.055219] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5347.545310] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5347.621696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5349.024775] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5349.110875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5351.906778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.907904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5351.908396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5351.908400] Lustre: Skipped 8 previous similar messages [ 5351.913565] Lustre: Skipped 1 previous similar message [ 5364.191095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5364.279645] Lustre: server umount lustre-OST0000 complete [ 5372.591386] Lustre: server umount lustre-MDT0000 complete [ 5374.485868] LustreError: 211177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852512 with bad export cookie 9383007466056992379 [ 5374.489905] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.493312] LustreError: 211177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5374.631296] Lustre: server umount lustre-MDT0001 complete [ 5376.432225] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:28:33 (1788852513) [ 5384.924721] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 5389.613689] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5406.959304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.060219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5407.075806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5407.115366] Lustre: lustre-MDT0000: new disk, initializing [ 5407.166045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5409.562096] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5414.944837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.999934] Lustre: 217758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5415.132228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5416.842303] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5421.384322] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5423.653115] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.320120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.466312] Lustre: 219726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5427.898846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5428.884776] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5433.246736] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5435.004547] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5435.090547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5436.740118] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5436.817962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5438.432971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.433107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.435252] LustreError: Skipped 2 previous similar messages [ 5438.435965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.440111] Lustre: Skipped 3 previous similar messages [ 5438.446830] Lustre: Skipped 10 previous similar messages [ 5443.738203] Lustre: server umount lustre-OST0000 complete [ 5451.631972] Lustre: server umount lustre-MDT0000 complete [ 5453.224521] LustreError: 217751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852591 with bad export cookie 9383007466056993261 [ 5453.227061] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5453.227148] LustreError: 190426:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8e50ccdeb800 x1875747456386304/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1788852600 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 5453.233247] LustreError: 217751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5453.421562] Lustre: server umount lustre-MDT0001 complete [ 5457.801961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.004539] LustreError: 221938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.016936] LustreError: 221938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5458.038930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.042655] Lustre: Skipped 5 previous similar messages [ 5459.700694] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5463.510923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.390203] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5468.598944] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5470.894513] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5474.011472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5477.555228] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5481.113510] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5498.848535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5498.851441] LustreError: Skipped 2 previous similar messages [ 5500.499760] Lustre: server umount lustre-OST0000 complete [ 5503.968609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.969935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.973127] Lustre: Skipped 5 previous similar messages [ 5503.975518] Lustre: Skipped 7 previous similar messages [ 5508.002045] Lustre: server umount lustre-MDT0000 complete [ 5509.433273] LustreError: 221919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852647 with bad export cookie 9383007466056994143 [ 5509.436945] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.438796] LustreError: 221919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5509.554766] Lustre: server umount lustre-MDT0001 complete [ 5615.446341] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5616.869986] Key type lgssc unregistered [ 5617.041630] LNet: 225615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.046977] LNetError: 225615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.056510] LNet: Removed LNI 192.168.206.118@tcp [ 5617.459176] Key type .llcrypt unregistered [ 5617.461481] Key type ._llcrypt unregistered [ 5627.344477] Key type ._llcrypt registered [ 5627.345786] Key type .llcrypt registered [ 5627.388234] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 5633.858285] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5634.233066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5634.279657] alg: No test for adler32 (adler32-zlib) [ 5635.178352] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 5635.300056] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5636.903270] Key type lgssc registered [ 5637.439943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5652.992234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5653.006669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.128326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5654.145672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5654.188210] Lustre: lustre-MDT0000: new disk, initializing [ 5654.226521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5654.234462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5656.124675] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5661.561894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.633238] Lustre: 228936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5661.674375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5661.677474] Lustre: Skipped 1 previous similar message [ 5661.726747] Lustre: lustre-MDT0001: new disk, initializing [ 5661.770737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5661.788665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5661.792546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5664.047668] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5669.975527] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5673.541278] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5677.696097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5677.866161] Lustre: lustre-OST0000: new disk, initializing [ 5677.869737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5677.876676] Lustre: 230903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5677.923502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.304794] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5683.208305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5683.213120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5683.267576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5686.004294] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5687.967703] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5688.101929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5689.960910] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5690.068658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5693.409813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5693.410689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.413199] LustreError: Skipped 1 previous similar message [ 5693.417883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5693.422581] Lustre: Skipped 1 previous similar message [ 5698.529489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5698.531923] Lustre: Skipped 1 previous similar message [ 5703.650480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.183192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5705.272678] Lustre: server umount lustre-OST0000 complete [ 5707.744291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5707.751807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.761953] Lustre: Skipped 1 previous similar message [ 5707.766905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.773108] Lustre: Skipped 1 previous similar message [ 5708.770228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.689842] Lustre: server umount lustre-MDT0000 complete [ 5715.656410] LustreError: 228927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852853 with bad export cookie 6153697884041401532 [ 5715.659798] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.674900] LustreError: 228927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5715.847243] Lustre: server umount lustre-MDT0001 complete [ 5724.597508] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5728.957634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.113698] LustreError: 233831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.122727] LustreError: 233831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5729.132530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5730.927701] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5734.368037] LustreError: 233832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.532781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.695468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.368715] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5737.660688] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5740.686274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5743.755395] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5748.127068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.216453] Lustre: lustre-OST0001: new disk, initializing [ 5748.218602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5748.222685] Lustre: 236137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5748.270825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5748.274292] Lustre: Skipped 1 previous similar message [ 5751.238772] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5754.375130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5754.379494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5754.415282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5761.734244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5767.580548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5773.043429] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:35:10 (1788852910) [ 5775.842831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5775.843176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5775.844656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5775.844663] Lustre: Skipped 2 previous similar messages [ 5775.854827] Lustre: Skipped 2 previous similar messages [ 5776.109448] Lustre: server umount lustre-MDT0000 complete [ 5777.709280] LustreError: 233814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852915 with bad export cookie 6153697884041402407 [ 5777.712147] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.714396] LustreError: 233814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5777.895486] Lustre: server umount lustre-MDT0001 complete [ 5789.188448] Lustre: server umount lustre-OST0000 complete [ 5800.988928] Lustre: server umount lustre-OST0001 complete [ 5803.405411] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 5807.071575] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5821.621760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5821.738045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5821.751548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5821.791197] Lustre: lustre-MDT0000: new disk, initializing [ 5821.821697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.830504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5823.806222] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5829.240742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5829.308723] Lustre: 240077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5829.483446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5832.178279] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5837.122378] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5839.572808] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5842.863296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5842.997370] Lustre: lustre-OST0000: new disk, initializing [ 5842.998920] Lustre: Skipped 1 previous similar message [ 5843.001089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5843.003321] Lustre: Skipped 2 previous similar messages [ 5843.006416] Lustre: 242042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5844.879925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5844.885268] Lustre: Skipped 1 previous similar message [ 5844.888602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5844.948861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5847.273874] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5853.723986] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5856.652236] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5856.824383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5859.775725] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5859.920704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5865.444936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.445625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5865.446199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.446204] Lustre: Skipped 3 previous similar messages [ 5865.451207] Lustre: Skipped 2 previous similar messages [ 5865.459074] LustreError: Skipped 1 previous similar message [ 5867.099645] Lustre: server umount lustre-OST0000 complete [ 5870.560743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5874.847422] Lustre: server umount lustre-MDT0000 complete [ 5875.680553] LustreError: 240083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.690782] LustreError: 240083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5876.302537] LustreError: 240067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853014 with bad export cookie 6153697884041404528 [ 5876.305548] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5876.306707] LustreError: 240067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5876.423602] Lustre: server umount lustre-MDT0001 complete [ 5879.663060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.878069] LustreError: 244257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.931559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.935139] Lustre: Skipped 2 previous similar messages [ 5881.433512] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5884.908955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.948605] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5889.827635] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5892.007466] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5894.935249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5898.236371] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5901.557605] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6047.719207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.719507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6047.725214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6047.725221] Lustre: Skipped 3 previous similar messages [ 6047.728788] Lustre: Skipped 3 previous similar messages [ 6052.290598] Lustre: server umount lustre-OST0000 complete [ 6060.571877] Lustre: server umount lustre-MDT0000 complete [ 6062.945384] LustreError: 246099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853201 with bad export cookie 6153697884041405403 [ 6062.948195] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.954604] LustreError: 246099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6063.155400] Lustre: server umount lustre-MDT0001 complete [ 6168.515823] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6169.940969] Key type lgssc unregistered [ 6170.122703] LNet: 247865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6170.125819] LNetError: 247865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6170.133845] LNet: Removed LNI 192.168.206.118@tcp [ 6170.647550] Key type .llcrypt unregistered [ 6170.649054] Key type ._llcrypt unregistered [ 6182.033626] Key type ._llcrypt registered [ 6182.037957] Key type .llcrypt registered [ 6182.092307] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 6190.074664] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6190.441244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6190.486095] alg: No test for adler32 (adler32-zlib) [ 6191.369664] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 6191.472907] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6193.071281] Key type lgssc registered [ 6193.536916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6215.560229] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6220.272769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6220.286023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.396151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6221.413808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6221.460154] Lustre: lustre-MDT0000: new disk, initializing [ 6221.506648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6221.520868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6223.340617] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6228.776662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.829721] Lustre: 251900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6228.867542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6228.871720] Lustre: Skipped 1 previous similar message [ 6228.917704] Lustre: lustre-MDT0001: new disk, initializing [ 6228.965207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6228.984590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6228.988978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6231.124365] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6234.010793] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6237.277624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.407254] Lustre: lustre-OST0000: new disk, initializing [ 6237.410118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6237.415085] Lustre: 253529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6237.457592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.283817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6238.288163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6238.368041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6240.275173] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6246.281617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6246.374081] Lustre: lustre-OST0001: new disk, initializing [ 6246.376802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6246.380970] Lustre: 254402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6246.436636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6248.357278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6248.362612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6248.418195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6249.716615] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6257.128125] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6262.364523] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6269.921692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6269.925623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.937761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6271.405392] Lustre: server umount lustre-MDT0000 complete [ 6273.571877] LustreError: 253162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853411 with bad export cookie 14317076585617786102 [ 6273.576413] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.017647] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.019303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6274.030160] Lustre: Skipped 3 previous similar messages [ 6274.032882] Lustre: Skipped 1 previous similar message [ 6275.741608] Lustre: server umount lustre-MDT0001 complete [ 6280.358250] Lustre: server umount lustre-OST0000 complete [ 6282.702934] Lustre: server umount lustre-OST0001 complete [ 6291.492588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6297.461898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6302.471837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6308.240519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6318.923026] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6323.585929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.597226] Lustre: lustre-MDT0000: reset Object Index mappings [ 6323.823068] LustreError: 259400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6323.834141] LustreError: 259400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6323.871675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.583499] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6329.173016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.181273] Lustre: lustre-MDT0001: reset Object Index mappings [ 6331.192832] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6332.430161] Lustre: 260543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6335.400679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6335.622859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.626523] Lustre: Skipped 1 previous similar message [ 6338.525485] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6339.747943] LustreError: 260896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.712775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6345.641640] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6347.954507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6347.965413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6354.901625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6356.541891] Lustre: 262413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6360.032428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6360.035820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.042070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.617643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.617688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.625066] Lustre: Skipped 1 previous similar message [ 6363.630649] Lustre: Skipped 3 previous similar messages [ 6364.736307] Lustre: server umount lustre-MDT0000 complete [ 6366.542857] LustreError: 260545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853504 with bad export cookie 14317076585617792871 [ 6366.544428] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6366.547945] LustreError: 260545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6366.744342] Lustre: server umount lustre-MDT0001 complete [ 6378.972601] Lustre: server umount lustre-OST0000 complete [ 6390.056100] Lustre: server umount lustre-OST0001 complete [ 6398.709834] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:45:36 (1788853536) [ 6405.944450] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 6410.249105] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6429.281440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.418013] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6429.434370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6429.474914] Lustre: lustre-MDT0000: new disk, initializing [ 6429.504516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.507178] Lustre: Skipped 1 previous similar message [ 6429.515426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6431.451422] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6437.493048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.560704] Lustre: 266664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6437.600661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6437.606114] Lustre: Skipped 1 previous similar message [ 6437.654246] Lustre: lustre-MDT0001: new disk, initializing [ 6437.717950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6437.724858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6439.486526] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6444.203288] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6446.497597] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6449.459638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.599390] Lustre: 268632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6451.267707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6451.314956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6452.261527] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6457.463930] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6459.801613] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6459.926601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6462.096477] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6462.198122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6466.529038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6466.531949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.537422] Lustre: Skipped 1 previous similar message [ 6466.539734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6469.364502] Lustre: server umount lustre-OST0000 complete [ 6477.281371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.285109] Lustre: Skipped 5 previous similar messages [ 6477.845449] Lustre: server umount lustre-MDT0000 complete [ 6479.913223] LustreError: 266657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853617 with bad export cookie 14317076585617794957 [ 6479.918318] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.921070] LustreError: 266657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6480.107158] Lustre: server umount lustre-MDT0001 complete [ 6490.737762] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6496.183637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.392071] LustreError: 271561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.421278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.427102] Lustre: Skipped 2 previous similar messages [ 6498.565644] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6501.857650] LustreError: 271562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6502.449307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.165909] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6505.285900] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6507.720338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.333539] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6513.879289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.961647] Lustre: lustre-OST0001: new disk, initializing [ 6513.963330] Lustre: Skipped 1 previous similar message [ 6513.965430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6513.967831] Lustre: Skipped 1 previous similar message [ 6513.970646] Lustre: 273869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6515.115761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6515.122063] Lustre: Skipped 1 previous similar message [ 6515.126598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6515.174716] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6516.689821] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6522.503765] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6524.343435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6526.957668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6527.005617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6532.474684] Lustre: server umount lustre-MDT0000 complete [ 6533.601761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6533.606170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.611311] Lustre: Skipped 3 previous similar messages [ 6533.614148] LustreError: 272288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.349191] LustreError: 271541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853672 with bad export cookie 14317076585617795846 [ 6534.354693] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6534.357189] LustreError: 271541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6536.674932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6536.677710] Lustre: Skipped 1 previous similar message [ 6540.582948] Lustre: server umount lustre-MDT0001 complete [ 6542.461218] Lustre: server umount lustre-OST0000 complete [ 6544.399517] Lustre: server umount lustre-OST0001 complete [ 6551.556329] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6556.021312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.155045] LustreError: 277204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.166111] LustreError: 277204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6557.850148] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6561.304520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.478536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6561.481952] Lustre: Skipped 4 previous similar messages [ 6563.111464] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6564.210068] Lustre: 278341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6566.935071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6569.646252] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6573.490130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6575.971114] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6581.746171] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6581.773231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6584.824215] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6586.480620] Lustre: 280220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6588.724113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6588.741984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6595.137178] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:48:52 (1788853732) [ 6607.841051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6607.844045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.849881] Lustre: Skipped 4 previous similar messages [ 6607.852046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6607.853697] Lustre: Skipped 2 previous similar messages [ 6614.006549] Lustre: server umount lustre-MDT0000 complete [ 6616.256620] LustreError: 277184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853754 with bad export cookie 14317076585617802223 [ 6616.263168] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.273633] LustreError: 277184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6617.568202] LustreError: 277199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.579383] LustreError: 277199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6622.481666] Lustre: server umount lustre-MDT0001 complete [ 6630.547100] Lustre: server umount lustre-OST0000 complete [ 6638.588796] Lustre: server umount lustre-OST0001 complete [ 6641.954420] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 6647.564210] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6669.384277] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6673.797729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.901923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6673.918329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6673.959526] Lustre: lustre-MDT0000: new disk, initializing [ 6674.009315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6676.160401] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6681.508814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.565843] Lustre: 284746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6681.691496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6683.406488] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6689.227562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6689.349689] Lustre: 286379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6689.524136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6691.679713] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6696.476397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.597870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.601256] Lustre: Skipped 5 previous similar messages [ 6698.417421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6699.092501] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6704.252161] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6711.741610] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:50:49 (1788853849) [ 6713.832212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.834190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6713.838520] Lustre: Skipped 8 previous similar messages [ 6713.844787] Lustre: Skipped 11 previous similar messages [ 6719.059214] Lustre: server umount lustre-MDT0000 complete [ 6721.262456] LustreError: 285541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853859 with bad export cookie 14317076585617810154 [ 6721.267088] LustreError: 285541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6721.267157] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6721.437671] Lustre: server umount lustre-MDT0001 complete [ 6734.368844] Lustre: server umount lustre-OST0000 complete [ 6746.802025] Lustre: server umount lustre-OST0001 complete [ 6750.849355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.080495] LustreError: 289489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.090660] LustreError: 289489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6752.884331] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6756.291157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.198311] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6761.507798] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.911306] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.223281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6770.105343] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6773.581229] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6790.757793] loop1: detected capacity change from 0 to 204800000 [ 6790.852815] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6790.860291] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6790.869725] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6793.350880] loop1: detected capacity change from 0 to 204800000 [ 6795.982347] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.031791] LustreError: 289476:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1448: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6796.040525] LustreError: 289476:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1448 [ 6796.043847] LustreError: 289476:0:(mgs_handler.c:517:mgs_target_reg()) fs1448-MDT0001: Error checking target index: rc = -13 [ 6796.051893] LustreError: fs1448-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6796.056794] LustreError: 293224:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 6796.061124] LustreError: 293224:0:(tgt_mount.c:2128:server_put_super()) no obd fs1448-MDT0001 [ 6796.064380] LustreError: 293224:0:(tgt_mount.c:116:server_deregister_mount()) fs1448-MDT0001 not registered [ 6796.090442] Lustre: server umount fs1448-MDT0001 complete [ 6796.093292] LustreError: 293224:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6800.088352] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.141070] Lustre: Setting parameter fs1448-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1448-MDT0001 [ 6800.146354] Lustre: Skipped 2 previous similar messages [ 6800.168389] Lustre: srv-fs1448-MDT0001: No data found on store. Initialize space. [ 6800.171313] Lustre: Skipped 4 previous similar messages [ 6800.218519] Lustre: fs1448-MDT0001: new disk, initializing [ 6800.220883] Lustre: Skipped 3 previous similar messages [ 6800.269994] Lustre: srv-fs1448-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6800.274678] Lustre: 293692:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1448-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6800.280113] LustreError: 293692:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6802.049213] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6805.661158] Lustre: server umount fs1448-MDT0001 complete [ 6811.525745] loop1: detected capacity change from 0 to 204800000 [ 6814.002949] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.161645] Lustre: 295257:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1448-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6814.165711] LustreError: 295257:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6815.862763] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6817.762343] LustreError: 291318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.769557] LustreError: 291318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6817.834352] Lustre: server umount fs1448-MDT0001 complete [ 6822.881608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.881862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6828.143979] Lustre: server umount lustre-MDT0000 complete [ 6831.417020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.490233] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6833.356457] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6836.381075] loop1: detected capacity change from 0 to 204800000 [ 6836.711419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6838.554990] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.697602] Lustre: 297699:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1448-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6838.705552] LustreError: 297699:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6840.316654] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6842.392721] Lustre: server umount fs1448-MDT0001 complete [ 6852.066149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6852.070730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6852.075191] Lustre: Skipped 12 previous similar messages [ 6865.887412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6865.959727] Lustre: server umount lustre-OST0000 complete [ 6873.793895] Lustre: server umount lustre-MDT0000 complete [ 6875.380273] LustreError: 289470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854013 with bad export cookie 14317076585617812835 [ 6875.382963] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6875.385822] LustreError: 289470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6875.538649] Lustre: server umount lustre-MDT0001 complete [ 6879.942456] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6881.278524] Key type lgssc unregistered [ 6881.449512] LNet: 300039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6881.453582] LNetError: 300039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6881.466067] LNet: Removed LNI 192.168.206.118@tcp [ 6881.842131] Key type .llcrypt unregistered [ 6881.843862] Key type ._llcrypt unregistered [ 6890.985655] Key type ._llcrypt registered [ 6890.987161] Key type .llcrypt registered [ 6891.038524] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6891.462385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6891.474382] alg: No test for adler32 (adler32-zlib) [ 6892.373848] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 6892.488550] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6894.087404] Key type lgssc registered [ 6894.552708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6897.938911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6897.956417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.201921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.860930] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6904.273715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.484091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6906.142946] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6909.212736] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6911.968919] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6915.569182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.793977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.650582] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6922.268474] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.009053] LustreError: 303125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.024701] LustreError: 303125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6929.127674] LustreError: 303798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.136482] LustreError: 303798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6934.498598] LustreError: 303125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6934.520532] LustreError: 303125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6938.353203] loop1: detected capacity change from 0 to 204800000 [ 6938.419831] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6938.425822] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6938.433914] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6939.619653] LustreError: 303798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.627699] LustreError: 303798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6940.751730] loop1: detected capacity change from 0 to 204800000 [ 6943.010392] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6943.078901] Lustre: fs18743-OST0000: new disk, initializing [ 6943.081959] Lustre: srv-fs18743-OST0000: No data found on store. Initialize space. [ 6943.086220] Lustre: 305032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6943.121736] Lustre: fs18743-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6944.740691] LustreError: 303128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.748681] LustreError: 303128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6945.638517] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6949.886967] Lustre: server umount fs18743-OST0000 complete [ 6954.980232] LustreError: 303128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.988873] LustreError: 303128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6955.090239] loop1: detected capacity change from 0 to 204800000 [ 6957.680385] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.761141] Lustre: fs18743-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.255784] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6964.196534] loop2: detected capacity change from 0 to 204800000 [ 6964.285125] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6964.290833] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6964.297723] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6966.730799] loop2: detected capacity change from 0 to 204800000 [ 6969.375869] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6969.412951] LustreError: fs18743-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6969.418954] LustreError: fs18743-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6969.424174] LustreError: 308066:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 6969.429635] LustreError: 308066:0:(tgt_mount.c:2128:server_put_super()) no obd fs18743-OST0001 [ 6969.432447] LustreError: 308066:0:(tgt_mount.c:116:server_deregister_mount()) fs18743-OST0001 not registered [ 6969.450797] Lustre: server umount fs18743-OST0001 complete [ 6969.452271] LustreError: 308066:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6971.046078] LustreError: 303798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.052491] LustreError: 303798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 6973.715787] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.766889] Lustre: fs18743-OST0001: new disk, initializing [ 6973.769254] Lustre: srv-fs18743-OST0001: No data found on store. Initialize space. [ 6973.772536] Lustre: 308618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6973.801652] Lustre: fs18743-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6976.160742] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6980.527304] Lustre: server umount fs18743-OST0000 complete [ 6984.370528] Lustre: server umount fs18743-OST0001 complete [ 6989.777122] loop1: detected capacity change from 0 to 204800000 [ 6992.420101] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6992.506413] Lustre: fs18743-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.544411] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6999.292528] loop2: detected capacity change from 0 to 204800000 [ 7002.359559] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7005.984453] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7006.184422] LustreError: 303126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.191498] LustreError: 303126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 7010.061067] Lustre: server umount fs18743-OST0000 complete [ 7011.306175] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7014.989586] Lustre: server umount fs18743-OST0001 complete [ 7016.423555] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7021.541607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.541692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.550671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.553040] LustreError: Skipped 1 previous similar message [ 7026.599581] Lustre: server umount lustre-OST0000 complete [ 7031.780808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.783341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7031.790290] Lustre: Skipped 2 previous similar messages [ 7031.794736] Lustre: Skipped 1 previous similar message [ 7034.887687] Lustre: server umount lustre-MDT0000 complete [ 7036.984619] LustreError: 301278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854175 with bad export cookie 5909362711796670458 [ 7036.986078] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.990061] LustreError: 301278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7037.173421] Lustre: server umount lustre-MDT0001 complete [ 7042.287032] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7043.707616] Key type lgssc unregistered [ 7043.900506] LNet: 314531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7043.906749] LNetError: 314531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7044.972596] LNet: Removed LNI 192.168.206.118@tcp [ 7045.482321] Key type .llcrypt unregistered [ 7045.484946] Key type ._llcrypt unregistered [ 7063.401130] Key type ._llcrypt registered [ 7063.403473] Key type .llcrypt registered [ 7063.469893] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7071.515765] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7071.890020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7071.904292] alg: No test for adler32 (adler32-zlib) [ 7072.812373] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7072.928609] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7074.527297] Key type lgssc registered [ 7075.033801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7092.068382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7092.080792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.209952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7093.224299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7093.268073] Lustre: lustre-MDT0000: new disk, initializing [ 7093.316349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.328665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7095.075627] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7100.282792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.359199] Lustre: 318359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7100.403167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7100.409201] Lustre: Skipped 1 previous similar message [ 7100.457639] Lustre: lustre-MDT0001: new disk, initializing [ 7100.514763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7100.533655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7100.538384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7102.514807] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7107.044144] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7109.319257] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7112.094648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7112.222278] Lustre: lustre-OST0000: new disk, initializing [ 7112.225142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7112.230306] Lustre: 320326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7112.268290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.564473] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7119.242720] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7120.653558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7120.660782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7120.707890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7126.016412] Lustre: Mounted lustre-client - version 2.17.58_39_g86006df [ 7126.566539] Lustre: Unmounted lustre-client [ 7133.347669] Lustre: server umount lustre-OST0000 complete [ 7136.225198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7136.225258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.226348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.234767] Lustre: Skipped 1 previous similar message [ 7141.344902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.348020] Lustre: Skipped 2 previous similar messages [ 7145.488978] Lustre: server umount lustre-MDT0000 complete [ 7147.329486] LustreError: 320325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854285 with bad export cookie 6322187726776162202 [ 7147.335038] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.339973] LustreError: 320325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7157.814195] Lustre: server umount lustre-MDT0001 complete [ 7163.351446] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7164.877439] Key type lgssc unregistered [ 7177.376507] LNet: 1 local NIs in recovery (showing 1): 192.168.206.118@tcp [ 7178.335689] LNet: 322486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7178.339773] LNetError: 322486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7251.045388] LNet: Removed LNI 192.168.206.118@tcp [ 7251.431828] Key type .llcrypt unregistered [ 7251.433143] Key type ._llcrypt unregistered [ 7261.036953] Key type ._llcrypt registered [ 7261.039664] Key type .llcrypt registered [ 7261.122343] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7270.010140] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7270.508410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7270.605546] alg: No test for adler32 (adler32-zlib) [ 7271.532126] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7271.669110] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7273.279190] Key type lgssc registered [ 7273.857414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7287.344979] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:00:24 (1788854424) [ 7290.242866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7290.255204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7291.367610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7291.382718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7291.434947] Lustre: lustre-MDT0000: new disk, initializing [ 7291.473920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.485894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7293.352679] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7298.397494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.449636] Lustre: 325677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7298.479908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7298.483801] Lustre: Skipped 1 previous similar message [ 7298.526104] Lustre: lustre-MDT0001: new disk, initializing [ 7298.568648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7298.585605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7298.589523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7300.399079] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7304.835722] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7307.577492] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7311.147494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7311.332707] Lustre: lustre-OST0000: new disk, initializing [ 7311.336528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7311.341759] Lustre: 327646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7311.399558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.649491] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7317.015280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7317.024457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7317.073485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7319.802978] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.454278] Lustre: server umount lustre-OST0000 complete [ 7332.321050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.321591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.326561] Lustre: Skipped 1 previous similar message [ 7337.440216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7337.443223] Lustre: Skipped 2 previous similar messages [ 7341.567644] Lustre: server umount lustre-MDT0000 complete [ 7343.312883] LustreError: 325669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854481 with bad export cookie 10002695345388399372 [ 7343.317732] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7353.937307] Lustre: server umount lustre-MDT0001 complete [ 7360.014934] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7361.708792] Key type lgssc unregistered [ 7361.905589] LNet: 329562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7361.911729] LNetError: 329562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7361.927106] LNet: Removed LNI 192.168.206.118@tcp [ 7362.473423] Key type .llcrypt unregistered [ 7362.475482] Key type ._llcrypt unregistered [ 7374.208454] Key type ._llcrypt registered [ 7374.209646] Key type .llcrypt registered [ 7374.253607] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7374.619572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.663232] alg: No test for adler32 (adler32-zlib) [ 7375.558319] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7375.663291] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7377.263243] Key type lgssc registered [ 7377.809852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7381.608188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7381.619373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.861241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.950296] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7389.501589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7389.718471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7391.785066] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7395.245253] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7397.863986] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7400.954412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.147447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.993487] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7407.743755] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7422.244727] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7422.889437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7424.690165] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 7424.992963] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.002412] Lustre: Skipped 2 previous similar messages [ 7429.798489] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 7429.802717] Lustre: Skipped 3 previous similar messages [ 7430.687470] Lustre: server umount lustre-MDT0000 complete [ 7432.370429] LustreError: 330790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854570 with bad export cookie 15578178425196002 [ 7432.373082] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7432.376108] LustreError: 330790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7434.918106] LustreError: 331903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.931669] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.18@tcp (stopping) [ 7434.934695] LustreError: 331903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7434.941960] Lustre: Skipped 3 previous similar messages [ 7435.232893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.280660] LustreError: 332647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.281468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7437.290179] LustreError: 332647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7437.293235] Lustre: Skipped 1 previous similar message [ 7438.619943] Lustre: server umount lustre-MDT0001 complete [ 7446.448431] Lustre: server umount lustre-OST0000 complete [ 7450.162091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7451.872917] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7453.106816] Lustre: 335006:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.118@tcp [ 7454.143106] Lustre: server umount MGS complete [ 7458.006487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.270782] LustreError: 335577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.308432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.379719] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7463.393513] LustreError: 335578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.667682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7464.862602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7466.721179] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7471.102624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7474.692846] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7478.195856] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7479.021737] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7481.795450] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7486.826580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7490.016955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7490.023205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.034396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7494.116516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.360405] Lustre: server umount lustre-OST0000 complete [ 7499.233441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.245154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7499.251514] Lustre: Skipped 3 previous similar messages [ 7504.037193] Lustre: server umount lustre-MDT0000 complete [ 7504.351929] LustreError: 335577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.369675] LustreError: 335577:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7506.475837] LustreError: 338259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854644 with bad export cookie 15578178425196828 [ 7506.482487] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.482901] LustreError: 338259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7506.683448] Lustre: server umount lustre-MDT0001 complete [ 7512.802417] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7514.581501] Key type lgssc unregistered [ 7514.786537] LNet: 339008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7514.793351] LNetError: 339008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7515.817194] LNet: Removed LNI 192.168.206.118@tcp [ 7516.382947] Key type .llcrypt unregistered [ 7516.384606] Key type ._llcrypt unregistered [ 7525.801585] Key type ._llcrypt registered [ 7525.803271] Key type .llcrypt registered [ 7525.878668] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7534.711431] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7535.144232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7535.207556] alg: No test for adler32 (adler32-zlib) [ 7536.149894] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7536.304582] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7537.927899] Key type lgssc registered [ 7538.569093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7557.340828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7557.363670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.471396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7558.484179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7558.534162] Lustre: lustre-MDT0000: new disk, initializing [ 7558.568991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7558.579988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7560.711719] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7566.639934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.704736] Lustre: 342141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7566.740266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7566.743654] Lustre: Skipped 1 previous similar message [ 7566.786594] Lustre: lustre-MDT0001: new disk, initializing [ 7566.829590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7566.852093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7566.857982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7569.208980] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7574.786968] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7578.099672] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7582.469581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7582.626397] Lustre: lustre-OST0000: new disk, initializing [ 7582.629542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7582.634773] Lustre: 344108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7582.683775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.565778] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7590.948771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7590.961939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7591.069405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7592.616964] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7594.998794] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7595.103663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7597.418565] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7597.541610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7601.122273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7601.131217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.139397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.633978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7601.644429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.656110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7604.835686] Lustre: server umount lustre-OST0000 complete [ 7606.752697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.754164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7606.758621] Lustre: Skipped 1 previous similar message [ 7611.872688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7611.875139] Lustre: Skipped 2 previous similar messages [ 7612.565715] Lustre: server umount lustre-MDT0000 complete [ 7614.090054] LustreError: 342939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854752 with bad export cookie 3498219318264928365 [ 7614.090825] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7614.095024] LustreError: 342939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7614.230576] Lustre: server umount lustre-MDT0001 complete [ 7617.738583] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:05:55 (1788854755) [ 7624.981893] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7629.440462] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7647.123588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.258694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7647.279349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7647.320668] Lustre: lustre-MDT0000: new disk, initializing [ 7647.370529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.407296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7649.611399] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7656.159868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.266573] Lustre: 348857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7656.299776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7656.307473] Lustre: Skipped 1 previous similar message [ 7656.354743] Lustre: lustre-MDT0001: new disk, initializing [ 7656.409611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7656.433641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7656.437590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7658.870337] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7665.283302] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7668.223821] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7671.532051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7671.689848] Lustre: lustre-OST0000: new disk, initializing [ 7671.692523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7671.697318] Lustre: 350824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7671.741649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.742420] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7679.505056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7679.510109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7679.569201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7679.651178] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7699.937072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7699.937632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.939970] LustreError: Skipped 1 previous similar message [ 7699.940692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7699.945690] Lustre: Skipped 1 previous similar message [ 7701.561981] Lustre: server umount lustre-OST0000 complete [ 7705.058117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.065841] Lustre: Skipped 1 previous similar message [ 7709.447592] Lustre: server umount lustre-MDT0000 complete [ 7711.466566] LustreError: 349652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854849 with bad export cookie 3498219318264929233 [ 7711.469566] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.477781] LustreError: 349652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7711.628629] Lustre: server umount lustre-MDT0001 complete [ 7717.012064] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7718.616694] Key type lgssc unregistered [ 7718.831671] LNet: 352969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7718.837891] LNetError: 352969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7718.847444] LNet: Removed LNI 192.168.206.118@tcp [ 7719.349104] Key type .llcrypt unregistered [ 7719.350368] Key type ._llcrypt unregistered [ 7729.659101] Key type ._llcrypt registered [ 7729.661207] Key type .llcrypt registered [ 7729.718469] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7730.269212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7730.282778] alg: No test for adler32 (adler32-zlib) [ 7731.215595] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7731.352797] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7732.960584] Key type lgssc registered [ 7733.472902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7737.074742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7737.086890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.274801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.098211] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7744.071439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.272361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.060118] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7749.247094] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7751.695578] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7754.390950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.524711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.911225] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7760.154225] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7789.538746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.543451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.543957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.548501] Lustre: Skipped 1 previous similar message [ 7792.689343] Lustre: server umount lustre-OST0000 complete [ 7795.680368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7795.683782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.689635] Lustre: Skipped 1 previous similar message [ 7795.692947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.777417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.778111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7801.026125] Lustre: server umount lustre-MDT0000 complete [ 7803.198679] LustreError: 354196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854941 with bad export cookie 1654222432984205111 [ 7803.202524] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7803.210445] LustreError: 354196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7803.338627] Lustre: server umount lustre-MDT0001 complete [ 7809.195763] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7810.652587] Key type lgssc unregistered [ 7810.809914] LNet: 359817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7810.813947] LNetError: 359817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7810.822778] LNet: Removed LNI 192.168.206.118@tcp [ 7811.280966] Key type .llcrypt unregistered [ 7811.283296] Key type ._llcrypt unregistered [ 7855.647816] Key type ._llcrypt registered [ 7855.649563] Key type .llcrypt registered [ 7855.695396] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7855.832536] Key type .llcrypt unregistered [ 7855.834653] Key type ._llcrypt unregistered [ 7889.847284] Key type ._llcrypt registered [ 7889.848606] Key type .llcrypt registered [ 7889.895458] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7890.002371] Key type .llcrypt unregistered [ 7890.003418] Key type ._llcrypt unregistered [ 7902.319481] Key type ._llcrypt registered [ 7902.320744] Key type .llcrypt registered [ 7902.370855] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7902.504269] Key type .llcrypt unregistered [ 7902.506298] Key type ._llcrypt unregistered [ 7914.657322] Key type ._llcrypt registered [ 7914.658682] Key type .llcrypt registered [ 7914.704208] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7914.813501] Key type .llcrypt unregistered [ 7914.815046] Key type ._llcrypt unregistered [ 7948.237122] Key type ._llcrypt registered [ 7948.238844] Key type .llcrypt registered [ 7948.298304] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7948.455444] Key type .llcrypt unregistered [ 7948.457274] Key type ._llcrypt unregistered [ 7962.380312] Key type ._llcrypt registered [ 7962.381779] Key type .llcrypt registered [ 7962.428898] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7962.550472] Key type .llcrypt unregistered [ 7962.552023] Key type ._llcrypt unregistered [ 7975.465743] Key type ._llcrypt registered [ 7975.467105] Key type .llcrypt registered [ 7975.510859] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7975.631299] Key type .llcrypt unregistered [ 7975.632808] Key type ._llcrypt unregistered [ 7990.348323] Key type ._llcrypt registered [ 7990.349949] Key type .llcrypt registered [ 7990.396908] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7990.527443] Key type .llcrypt unregistered [ 7990.528898] Key type ._llcrypt unregistered [ 8004.509087] Key type ._llcrypt registered [ 8004.510728] Key type .llcrypt registered [ 8004.561945] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8004.696423] Key type .llcrypt unregistered [ 8004.698150] Key type ._llcrypt unregistered [ 8017.805390] Key type ._llcrypt registered [ 8017.806877] Key type .llcrypt registered [ 8017.857915] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8018.010318] Key type .llcrypt unregistered [ 8018.011853] Key type ._llcrypt unregistered [ 8045.523249] Key type ._llcrypt registered [ 8045.524517] Key type .llcrypt registered [ 8045.580923] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8045.724766] Key type .llcrypt unregistered [ 8045.726225] Key type ._llcrypt unregistered [ 8058.472388] Key type ._llcrypt registered [ 8058.473610] Key type .llcrypt registered [ 8058.517935] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8058.646407] Key type .llcrypt unregistered [ 8058.648145] Key type ._llcrypt unregistered [ 8071.803601] Key type ._llcrypt registered [ 8071.804833] Key type .llcrypt registered [ 8071.852693] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8071.977263] Key type .llcrypt unregistered [ 8071.979372] Key type ._llcrypt unregistered [ 8096.258419] Key type ._llcrypt registered [ 8096.260285] Key type .llcrypt registered [ 8096.313803] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8104.049988] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8104.407085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8104.424707] alg: No test for adler32 (adler32-zlib) [ 8105.315471] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 8105.436355] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 8107.031550] Key type lgssc registered [ 8107.562693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8125.272336] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8125.289903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.403342] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8126.421029] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8126.461120] Lustre: tst250fs-MDT0000: new disk, initializing [ 8126.503165] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.514495] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8128.446921] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8134.312833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.391375] Lustre: 379671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8134.442883] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8134.445656] Lustre: Skipped 1 previous similar message [ 8134.496810] Lustre: tst250fs-MDT0001: new disk, initializing [ 8134.556465] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8134.579024] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8134.586897] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8136.793612] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8139.452483] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8142.220738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8142.340705] Lustre: tst250fs-OST0000: new disk, initializing [ 8142.344599] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8142.347686] Lustre: 381305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8142.383528] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.735965] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8149.523049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8149.598171] Lustre: tst250fs-OST0001: new disk, initializing [ 8149.600641] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8149.604057] Lustre: 382164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8149.652306] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.014861] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8150.020144] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8150.091436] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8150.100705] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8152.172958] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8155.222724] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8170.468025] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.469547] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8170.474568] Lustre: Skipped 1 previous similar message [ 8170.474947] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8170.478878] Lustre: Skipped 1 previous similar message [ 8172.579700] Lustre: server umount tst250fs-MDT0000 complete [ 8174.126328] LustreError: 379665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855312 with bad export cookie 16539683821696308979 [ 8174.130551] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8174.133882] LustreError: 379665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8174.278464] Lustre: server umount tst250fs-MDT0001 complete [ 8186.193145] Lustre: server umount tst250fs-OST0000 complete [ 8197.217739] Lustre: server umount tst250fs-OST0001 complete [ 8206.889121] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8210.994353] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8225.825138] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:16:03 (1788855363) [ 8226.589648] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8234.056837] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8239.077709] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8255.499608] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8028 sec ========== 04:16:32 (1788855392) [ 8256.412759] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:16:33 (1788855393) === [ 8258.197746] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:16:35 (1788855395) === [ 8276.846428] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8278.163332] Key type lgssc unregistered [ 8278.315592] LNet: 391280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8278.319709] LNetError: 391280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8278.328622] LNet: Removed LNI 192.168.206.118@tcp [ 8278.765177] Key type .llcrypt unregistered [ 8278.769263] Key type ._llcrypt unregistered