[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 325674114 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.521868] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.524815] pid_max: default: 32768 minimum: 301 [ 0.526520] Security Framework initialized [ 0.528121] SELinux: Initializing. [ 0.529443] SELinux: Starting in permissive mode [ 0.531265] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.536396] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.539836] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.542337] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.545320] Initializing cgroup subsys memory [ 0.546920] Initializing cgroup subsys devices [ 0.548653] Initializing cgroup subsys freezer [ 0.550390] Initializing cgroup subsys net_cls [ 0.552017] Initializing cgroup subsys blkio [ 0.553637] Initializing cgroup subsys perf_event [ 0.555296] Initializing cgroup subsys hugetlb [ 0.556937] Initializing cgroup subsys pids [ 0.558474] Initializing cgroup subsys net_prio [ 0.560480] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.563922] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.565854] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.567750] tlb_flushall_shift: 6 [ 0.569057] FEATURE SPEC_CTRL Present [ 0.570379] FEATURE IBPB_SUPPORT Present [ 0.571928] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.574249] Spectre V2 : Vulnerable [ 0.575529] Speculative Store Bypass: Vulnerable [ 0.578359] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.587447] ACPI: Core revision 20130517 [ 0.591041] ACPI: All ACPI Tables successfully acquired [ 0.593070] ftrace: allocating 30294 entries in 119 pages [ 0.651453] Enabling x2apic [ 0.652390] Enabled x2apic [ 0.653595] Switched APIC routing to physical x2apic. [ 0.657092] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.659143] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.662416] TSC deadline timer enabled [ 0.662584] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.665168] ... version: 2 [ 0.666276] ... bit width: 48 [ 0.667444] ... generic registers: 4 [ 0.668614] ... value mask: 0000ffffffffffff [ 0.670351] ... max period: 00007fffffffffff [ 0.671981] ... fixed-purpose events: 3 [ 0.673071] ... event mask: 000000070000000f [ 0.674574] KVM setup paravirtual spinlock [ 0.679748] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.683017] KVM setup async PF for cpu 1 [ 0.684328] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.686731] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.690384] KVM setup async PF for cpu 2 [ 0.677970] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.690880] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.694330] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.696029] Brought up 4 CPUs [ 0.696045] KVM setup async PF for cpu 3 [ 0.696053] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.699595] smpboot: Max logical packages: 1 [ 0.700759] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.704664] devtmpfs: initialized [ 0.705833] x86/mm: Memory block size: 128MB [ 0.710635] EVM: security.selinux [ 0.711630] EVM: security.ima [ 0.712536] EVM: security.capability [ 0.716289] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.718530] NET: Registered protocol family 16 [ 0.720126] cpuidle: using governor haltpoll [ 0.721739] ACPI: bus type PCI registered [ 0.722962] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.725147] PCI: Using configuration type 1 for base access [ 0.726848] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.735252] ACPI: Added _OSI(Module Device) [ 0.736384] ACPI: Added _OSI(Processor Device) [ 0.737555] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.738721] ACPI: Added _OSI(Processor Aggregator Device) [ 0.740187] ACPI: Added _OSI(Linux-Dell-Video) [ 0.743277] ACPI: EC: Look up EC in DSDT [ 0.746427] ACPI: Interpreter enabled [ 0.747464] ACPI: (supports S0 S3 S4 S5) [ 0.748459] ACPI: Using IOAPIC for interrupt routing [ 0.749742] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.752188] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.758160] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.759794] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.761562] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.763225] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.766705] acpiphp: Slot [2] registered [ 0.767711] acpiphp: Slot [3] registered [ 0.768795] acpiphp: Slot [4] registered [ 0.769941] acpiphp: Slot [5] registered [ 0.771013] acpiphp: Slot [6] registered [ 0.772101] acpiphp: Slot [7] registered [ 0.773186] acpiphp: Slot [8] registered [ 0.774329] acpiphp: Slot [9] registered [ 0.775456] acpiphp: Slot [10] registered [ 0.776594] acpiphp: Slot [11] registered [ 0.777718] acpiphp: Slot [12] registered [ 0.778823] acpiphp: Slot [13] registered [ 0.779925] acpiphp: Slot [14] registered [ 0.781056] acpiphp: Slot [15] registered [ 0.782161] acpiphp: Slot [16] registered [ 0.783235] acpiphp: Slot [17] registered [ 0.784311] acpiphp: Slot [18] registered [ 0.785414] acpiphp: Slot [19] registered [ 0.786544] acpiphp: Slot [20] registered [ 0.787638] acpiphp: Slot [21] registered [ 0.788729] acpiphp: Slot [22] registered [ 0.789859] acpiphp: Slot [23] registered [ 0.790944] acpiphp: Slot [24] registered [ 0.792023] acpiphp: Slot [25] registered [ 0.793109] acpiphp: Slot [26] registered [ 0.794204] acpiphp: Slot [27] registered [ 0.795224] acpiphp: Slot [28] registered [ 0.796475] acpiphp: Slot [29] registered [ 0.797547] acpiphp: Slot [30] registered [ 0.798629] acpiphp: Slot [31] registered [ 0.799654] PCI host bridge to bus 0000:00 [ 0.800674] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.802314] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.804006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.805860] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.807733] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.809605] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.811018] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.811711] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.812538] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.817555] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.819104] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.820997] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.822580] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.824358] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.826312] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.826868] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.829394] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.831684] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.833316] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.840499] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.841979] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.842905] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.848265] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.853582] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.864488] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.869110] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.872586] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.878733] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.896292] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.902190] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.905633] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.909729] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.921318] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.926310] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.931534] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.936029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.948688] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.953213] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.958004] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.961271] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.970517] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.974251] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.977292] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.985992] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.998315] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.002484] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.004086] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.007407] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.008626] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.009951] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.011658] vgaarb: loaded [ 1.012390] SCSI subsystem initialized [ 1.013252] ACPI: bus type USB registered [ 1.014158] usbcore: registered new interface driver usbfs [ 1.015928] usbcore: registered new interface driver hub [ 1.017510] usbcore: registered new device driver usb [ 1.019003] PCI: Using ACPI for IRQ routing [ 1.019826] PCI: pci_cache_line_size set to 64 bytes [ 1.020056] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.020060] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.020061] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.020371] NetLabel: Initializing [ 1.021101] NetLabel: domain hash size = 128 [ 1.021891] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.022826] NetLabel: unlabeled traffic allowed by default [ 1.024093] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.025040] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.030390] amd_nb: Cannot enumerate AMD northbridges [ 1.031509] Switched to clocksource kvm-clock [ 1.045444] pnp: PnP ACPI init [ 1.046209] ACPI: bus type PNP registered [ 1.047095] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.047163] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.047203] pnp 00:02: [dma 2] [ 1.047236] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.047324] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.047404] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.047494] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.047905] pnp: PnP ACPI: found 6 devices [ 1.049130] ACPI: bus type PNP unregistered [ 1.060435] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.060442] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.060444] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.060446] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.060448] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.060536] NET: Registered protocol family 2 [ 1.062224] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.064789] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.066914] TCP: Hash tables configured (established 32768 bind 32768) [ 1.068403] TCP: reno registered [ 1.069622] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.071528] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.073735] NET: Registered protocol family 1 [ 1.075690] RPC: Registered named UNIX socket transport module. [ 1.077404] RPC: Registered udp transport module. [ 1.078443] RPC: Registered tcp transport module. [ 1.079349] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.080693] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.082413] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084199] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.086017] PCI: CLS 0 bytes, default 64 [ 1.086809] Unpacking initramfs... [ 2.262317] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.266122] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.268297] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.271486] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.273416] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.274846] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.276173] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.278437] cryptomgr_test (51) used greatest stack depth: 14128 bytes left [ 2.278880] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.278923] Initialise system trusted keyring [ 2.305786] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.307260] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.311486] zpool: loaded [ 2.312155] zbud: loaded [ 2.313159] VFS: Disk quotas dquot_6.6.0 [ 2.313868] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.315442] NFS: Registering the id_resolver key type [ 2.316622] Key type id_resolver registered [ 2.317556] Key type id_legacy registered [ 2.318182] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.319525] Key type big_key registered [ 2.320141] SELinux: Registering netfilter hooks [ 2.321219] cryptomgr_test (57) used greatest stack depth: 14048 bytes left [ 2.323591] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.325512] NET: Registered protocol family 38 [ 2.326517] Key type asymmetric registered [ 2.327235] Asymmetric key parser 'x509' registered [ 2.328406] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.329925] io scheduler noop registered [ 2.330612] io scheduler deadline registered (default) [ 2.332251] io scheduler cfq registered [ 2.333387] io scheduler mq-deadline registered [ 2.334120] io scheduler kyber registered [ 2.336987] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.338472] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.339629] intel_idle: does not run on family 6 model 62 [ 2.339736] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.341385] ACPI: Power Button [PWRF] [ 2.342264] GHES: HEST is not enabled! [ 2.391733] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.444924] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.569271] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.635761] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.770231] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.797899] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.827416] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.830399] Non-volatile memory driver v1.3 [ 2.831872] Linux agpgart interface v0.103 [ 2.833050] crash memory driver: version 1.1 [ 2.834750] nbd: registered device at major 43 [ 2.838312] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.838339] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.838368] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.838391] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.838413] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.844620] virtio_blk virtio1: [vda] 66840 512-byte logical blocks (34.2 MB/32.6 MiB) [ 2.848700] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.848772] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.848805] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.848827] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.848850] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 2.855404] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.860352] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 2.860386] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 2.860419] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 2.860459] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 2.860495] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 2.865826] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.872136] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 2.872185] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 2.872223] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 2.872256] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 2.872289] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 2.878285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.886757] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 2.886826] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 2.886879] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 2.886918] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 2.886962] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 2.893466] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.899176] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 2.899314] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 2.899455] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 2.899514] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 2.899558] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 2.905960] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.912064] rdac: device handler registered [ 2.913714] hp_sw: device handler registered [ 2.914997] emc: device handler registered [ 2.916338] libphy: Fixed MDIO Bus: probed [ 2.920557] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.921617] ehci-pci: EHCI PCI platform driver [ 2.922927] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.924116] ohci-pci: OHCI PCI platform driver [ 2.925305] uhci_hcd: USB Universal Host Controller Interface driver [ 2.926703] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.930299] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.931894] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.933934] mousedev: PS/2 mouse device common for all mice [ 2.935903] rtc_cmos 00:05: RTC can wake from S4 [ 2.935974] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.941574] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 2.943924] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.947378] hidraw: raw HID events driver (C) Jiri Kosina [ 2.949264] usbcore: registered new interface driver usbhid [ 2.950937] usbhid: USB HID core driver [ 2.952001] drop_monitor: Initializing network drop monitor service [ 2.953900] Netfilter messages via NETLINK v0.30. [ 2.955392] TCP: cubic registered [ 2.956277] Initializing XFRM netlink socket [ 2.957780] NET: Registered protocol family 10 [ 2.959750] NET: Registered protocol family 17 [ 2.961122] Key type dns_resolver registered [ 2.962881] mce: Using 10 MCE banks [ 2.964150] Loading compiled-in X.509 certificates [ 2.966648] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 2.969552] registered taskstats version 1 [ 2.973230] modprobe (72) used greatest stack depth: 13456 bytes left [ 2.978075] Key type trusted registered [ 2.982837] Key type encrypted registered [ 2.983884] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 2.986818] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 2.989617] rtc_cmos 00:05: setting system clock to 2024-03-22 07:08:25 UTC (1711091305) [ 2.992756] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 2.994538] Write protecting the kernel read-only data: 12288k [ 2.995858] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 2.997079] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.005700] random: systemd: uninitialized urandom read (16 bytes read) [ 3.009168] random: systemd: uninitialized urandom read (16 bytes read) [ 3.011109] random: systemd: uninitialized urandom read (16 bytes read) [ 3.014898] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.020311] systemd[1]: Detected virtualization kvm. [ 3.021633] systemd[1]: Detected architecture x86-64. [ 3.022537] systemd[1]: Running in initial RAM disk. [ 3.025973] systemd[1]: No hostname configured. [ 3.027185] systemd[1]: Set hostname to . [ 3.028811] random: systemd: uninitialized urandom read (16 bytes read) [ 3.030896] systemd[1]: Initializing machine ID from random generator. [ 3.067482] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.069717] random: systemd: uninitialized urandom read (16 bytes read) [ 3.071144] random: systemd: uninitialized urandom read (16 bytes read) [ 3.073066] random: systemd: uninitialized urandom read (16 bytes read) [ 3.074647] random: systemd: uninitialized urandom read (16 bytes read) [ 3.077807] random: systemd: uninitialized urandom read (16 bytes read) [ 3.079874] random: systemd: uninitialized urandom read (16 bytes read) [ 3.090131] systemd[1]: Reached target Swap. [ 3.094233] systemd[1]: Created slice Root Slice. [ 3.098050] systemd[1]: Created slice System Slice. [ 3.101561] systemd[1]: Listening on udev Kernel Socket. [ 3.104997] systemd[1]: Reached target Timers. [ 3.108012] systemd[1]: Listening on udev Control Socket. [ 3.111472] systemd[1]: Reached target Slices. [ 3.114714] systemd[1]: Listening on Journal Socket. [ 3.120073] systemd[1]: Starting dracut cmdline hook... [ 3.123688] systemd[1]: Reached target Sockets. [ 3.128257] systemd[1]: Starting Journal Service... [ 3.133680] systemd[1]: Starting Setup Virtual Console... [ 3.139429] systemd[1]: Starting Load Kernel Modules... [ 3.146134] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.150083] systemd[1]: Reached target Local File Systems. [ 3.154341] systemd[1]: Started Setup Virtual Console. [ 3.160154] systemd[1]: Started Load Kernel Modules. [ 3.164333] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.170239] systemd[1]: Started Journal Service. [ 3.185946] random: fast init done [ 3.279597] tsc: Refined TSC clocksource calibration: 2399.952 MHz [ 3.615943] libata version 3.00 loaded. [ 3.617946] ata_piix 0000:00:01.1: version 2.13 [ 3.625785] scsi host0: ata_piix [ 3.632185] scsi host1: ata_piix [ 3.633675] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.635887] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.775172] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.828760] ip (321) used greatest stack depth: 13080 bytes left [ 3.872919] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.874476] ip (344) used greatest stack depth: 12464 bytes left [ 3.921630] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.930734] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.377996] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 7.670543] SELinux: Disabled at runtime. [ 7.671870] SELinux: Unregistering netfilter hooks [ 7.758071] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.761639] systemd[1]: Inserted module 'ip_tables' [ 8.280217] systemd-journald[566]: Received request to flush runtime journal from PID 1 [ 8.430340] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.464026] cryptd: max_cpu_qlen set to 1000 [ 8.467852] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.522478] AVX version of gcm_enc/dec engaged. [ 8.523949] AES CTR mode by8 optimization enabled [ 8.548575] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.563480] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.564579] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.565366] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.676022] EDAC MC: Ver: 3.0.0 [ 8.682435] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.682445] EDAC sbridge: Ver: 1.1.2 [ 10.762989] mount.nfs (772) used greatest stack depth: 10704 bytes left [ 17.790389] device-mapper: uevent: version 1.0.3 [ 17.791641] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 22.024322] libcfs: loading out-of-tree module taints kernel. [ 22.025774] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 26.956403] alg: No test for adler32 (adler32-zlib) [ 27.707628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 27.836578] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 27.991234] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 27.992697] LNet: Accept secure, port 988 [ 29.534722] Key type lgssc registered [ 29.810027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 32.583952] icp: module license 'CDDL' taints kernel. [ 32.585173] Disabling lock debugging due to kernel taint [ 35.194442] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 38.495016] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 42.985936] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 44.934442] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 44.937196] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 46.909354] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 46.912403] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 49.855948] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 52.843080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 52.870011] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 52.876000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 53.967580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 53.979112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 54.024726] Lustre: lustre-MDT0000: new disk, initializing [ 54.045291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 54.051698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 54.076782] mount.lustre (6522) used greatest stack depth: 9784 bytes left [ 54.822480] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 58.992223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.011465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 59.031823] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 59.040088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 59.042754] Lustre: Skipped 1 previous similar message [ 59.076487] Lustre: lustre-MDT0001: new disk, initializing [ 59.090836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 59.096715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 59.099813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 59.841400] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 64.415000] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 64.418431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.436813] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 64.440284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 64.514160] Lustre: lustre-OST0000: new disk, initializing [ 64.516128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 64.529900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 65.773810] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 69.227990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 69.230530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 69.244335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 70.110792] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 70.114335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.132881] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 70.136215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 70.164429] Lustre: lustre-OST0001: new disk, initializing [ 70.167159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 70.179543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 71.400867] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 74.432153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 74.434493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 74.442524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 76.224501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 80.580493] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 86.213141] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 86.608557] random: crng init done [ 87.033396] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 87.955161] Lustre: DEBUG MARKER: Client: 2.15.61.139 [ 88.540408] Lustre: DEBUG MARKER: MDS: 2.15.61.139 [ 89.126585] Lustre: DEBUG MARKER: OSS: 2.15.61.139 [ 89.480401] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Mar 22 03:09:51 EDT 2024 [ 90.961112] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 91.305253] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 104.156047] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 104.159421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 104.164334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 108.126418] Lustre: server umount lustre-MDT0000 complete [ 109.124109] LustreError: 6532:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091411 with bad export cookie 14002027703547774972 [ 109.125127] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 109.130724] LustreError: 6532:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 109.499911] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 109.500007] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 109.500009] Lustre: Skipped 3 previous similar messages [ 109.500123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 109.500125] Lustre: Skipped 3 previous similar messages [ 109.514994] LustreError: Skipped 1 previous similar message [ 114.508013] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 114.508014] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 114.508019] Lustre: Skipped 1 previous similar message [ 114.516269] LustreError: Skipped 1 previous similar message [ 115.233351] Lustre: server umount lustre-MDT0001 complete [ 122.335257] Lustre: server umount lustre-OST0000 complete [ 129.456525] Lustre: server umount lustre-OST0001 complete [ 130.850222] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 132.479181] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 135.050453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.816403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.437277] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 138.441692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.133707] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 140.137801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.055686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.077579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.157325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 142.166286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 142.206746] Lustre: lustre-MDT0000: new disk, initializing [ 142.227827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.232673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.084796] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 146.707912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.728208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.759594] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 146.846442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 147.672881] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 150.053605] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 150.676868] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 152.460284] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 152.464337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.483013] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 152.487086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 152.581950] Lustre: lustre-OST0000: new disk, initializing [ 152.583442] Lustre: Skipped 1 previous similar message [ 152.585858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 152.589195] Lustre: Skipped 2 previous similar messages [ 152.611127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.614434] Lustre: Skipped 1 previous similar message [ 153.609059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 153.612733] Lustre: Skipped 1 previous similar message [ 153.615462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 153.635528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 154.018736] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 156.408046] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 157.276483] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 157.329122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 158.151857] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 158.198573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 163.644527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.644877] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 163.645185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 163.645186] Lustre: Skipped 1 previous similar message [ 163.654978] Lustre: Skipped 2 previous similar messages [ 164.772611] Lustre: server umount lustre-OST0000 complete [ 166.860320] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 166.864087] LustreError: Skipped 1 previous similar message [ 166.865759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.652224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.655196] Lustre: Skipped 3 previous similar messages [ 171.821413] Lustre: server umount lustre-MDT0000 complete [ 172.815299] LustreError: 16881:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091475 with bad export cookie 14002027703547776414 [ 172.816456] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 172.822144] LustreError: 16881:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 172.930568] Lustre: server umount lustre-MDT0001 complete [ 175.159034] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 03:11:17 (1711091477) [ 176.921558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.070893] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.087986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.883636] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 180.179428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.096381] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 182.173639] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 182.777011] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 184.585558] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 184.589981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.975306] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 187.021585] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 192.388840] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 195.292190] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 195.295318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.300089] Lustre: Skipped 1 previous similar message [ 195.301905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 201.214991] Lustre: server umount lustre-MDT0000 complete [ 202.240311] LustreError: 21584:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091504 with bad export cookie 14002027703547777268 [ 202.241829] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 202.248246] LustreError: 21584:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 202.352841] Lustre: server umount lustre-MDT0001 complete [ 213.404676] Lustre: 23878:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091505/real 1711091505] req@ffff88012c0e2a00 x1794209302504960/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091515 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 213.490546] Lustre: server umount lustre-OST0000 complete [ 217.590657] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 219.747783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.871740] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.884357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.886485] Lustre: Skipped 2 previous similar messages [ 220.599338] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 222.628546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.463239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 225.561944] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 225.565682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.768563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 228.815879] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 228.818789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.839787] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 228.843241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.878643] Lustre: lustre-OST0001: new disk, initializing [ 228.880351] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 230.084193] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 230.638489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.642185] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.656548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.799419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.652288] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 242.668566] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 242.668829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 242.668830] Lustre: Skipped 3 previous similar messages [ 242.675977] Lustre: Skipped 4 previous similar messages [ 248.520027] Lustre: server umount lustre-MDT0000 complete [ 249.506458] LustreError: 28156:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091552 with bad export cookie 14002027703547778115 [ 249.508081] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.513275] LustreError: 28156:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 249.628318] Lustre: server umount lustre-MDT0001 complete [ 260.609717] Lustre: 30090:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091553/real 1711091553] req@ffff880085cd6680 x1794209302528192/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091563 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 260.640858] Lustre: server umount lustre-OST0000 complete [ 271.590652] Lustre: 30363:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091564/real 1711091564] req@ffff8800944b9f80 x1794209302528704/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091574 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 271.671120] Lustre: server umount lustre-OST0001 complete [ 275.043786] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 277.390057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.537256] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.341459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 280.526536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 281.407080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 283.651939] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 283.655716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.030966] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 287.236885] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 287.239901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.290079] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.291994] Lustre: Skipped 6 previous similar messages [ 288.575446] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 294.836379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.713268] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 295.715073] Lustre: Skipped 1 previous similar message [ 305.644184] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.646590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.649744] Lustre: Skipped 1 previous similar message [ 305.651438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.653848] Lustre: Skipped 7 previous similar messages [ 307.925255] Lustre: server umount lustre-MDT0000 complete [ 308.896623] LustreError: 2870:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091611 with bad export cookie 14002027703547779361 [ 308.898050] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.903303] LustreError: 2870:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.008592] Lustre: server umount lustre-MDT0001 complete [ 320.021657] Lustre: 3700:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091612/real 1711091612] req@ffff8801391d0a80 x1794209302546944/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091622 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 320.045387] Lustre: server umount lustre-OST0000 complete [ 331.087701] Lustre: 3988:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091623/real 1711091623] req@ffff880075465f80 x1794209302547456/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091633 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 331.159295] Lustre: server umount lustre-OST0001 complete [ 334.861903] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 03:13:56 (1711091636) [ 341.004680] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 343.311145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.443996] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.228544] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 346.453205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.365035] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 349.643706] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 349.646963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 350.999390] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 353.283069] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 353.286425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.626668] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 360.895299] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 361.834809] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 371.596319] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.599359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.605243] Lustre: Skipped 3 previous similar messages [ 371.606943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.608828] Lustre: Skipped 4 previous similar messages [ 373.834170] Lustre: server umount lustre-MDT0000 complete [ 374.899422] LustreError: 6810:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091677 with bad export cookie 14002027703547780432 [ 374.901093] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.908482] LustreError: 6810:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.012686] Lustre: server umount lustre-MDT0001 complete [ 386.073674] Lustre: 11121:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091678/real 1711091678] req@ffff88012de2aa00 x1794209302565440/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091688 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 386.097364] Lustre: server umount lustre-OST0000 complete [ 397.178328] Lustre: server umount lustre-OST0001 complete [ 400.943632] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 03:15:02 (1711091702) [ 404.136791] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 406.577701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.697681] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.705109] LustreError: Skipped 1 previous similar message [ 407.483268] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 409.767624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.634542] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 412.856008] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 412.859295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.194620] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 416.410250] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 416.414357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.468239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.471182] Lustre: Skipped 7 previous similar messages [ 417.696898] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 423.929363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 434.892124] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 436.884851] Lustre: server umount lustre-MDT0000 complete [ 437.866417] LustreError: 14306:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091740 with bad export cookie 14002027703547781503 [ 437.867471] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.873694] LustreError: 14306:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 440.508230] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.511790] LustreError: Skipped 2 previous similar messages [ 440.511919] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.511920] Lustre: Skipped 7 previous similar messages [ 443.990960] Lustre: server umount lustre-MDT0001 complete [ 446.140451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.516946] LustreError: 3104:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012d575c00 x1794209302585920/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 448.446468] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 450.715972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.594919] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 452.687198] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 454.098726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 454.661994] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 455.044248] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 462.019824] Lustre: server umount lustre-MDT0000 complete [ 463.095019] Lustre: server umount lustre-MDT0001 complete [ 470.106715] Lustre: server umount lustre-OST0000 complete [ 471.129045] Lustre: server umount lustre-OST0001 complete [ 474.789757] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 03:16:16 (1711091776) [ 477.825765] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 480.122471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.001635] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 483.199475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.028855] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 486.223096] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 486.228047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.556281] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 489.759923] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 489.763355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.061464] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 497.326534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 498.230408] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 498.232432] Lustre: Skipped 2 previous similar messages [ 507.853223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.855119] Lustre: Skipped 22 previous similar messages [ 512.881190] Lustre: server umount lustre-MDT0000 complete [ 513.876390] LustreError: 22672:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091816 with bad export cookie 14002027703547783393 [ 513.877839] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.877841] LustreError: Skipped 1 previous similar message [ 513.887196] LustreError: 22672:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 513.974951] Lustre: server umount lustre-MDT0001 complete [ 524.932658] Lustre: 26992:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091817/real 1711091817] req@ffff88012d576680 x1794209302617536/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091827 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 524.940424] Lustre: 26992:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 524.958258] Lustre: server umount lustre-OST0000 complete [ 535.953200] Lustre: server umount lustre-OST0001 complete [ 539.498764] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 03:17:21 (1711091841) [ 540.112598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 542.046251] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 03:17:24 (1711091844) [ 544.398815] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 545.782492] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 547.968359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.481528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 549.484651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.208770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.224267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.297678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 551.332850] Lustre: lustre-MDT0000: new disk, initializing [ 551.356476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 552.162459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 554.492528] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 556.197930] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 556.201082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 556.215516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 556.218809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.438531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 557.448410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 557.541187] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 559.798412] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 560.268538] Lustre: DEBUG MARKER: create test files [ 567.452066] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 572.916056] Lustre: server umount lustre-OST0000 complete [ 573.992230] Lustre: server umount lustre-MDT0000 complete [ 576.832895] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 577.293533] Key type lgssc unregistered [ 577.358116] LNet: 1091:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.360790] LNet: Removed LNI 192.168.204.114@tcp [ 585.265012] alg: No test for adler32 (adler32-zlib) [ 586.016601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 586.077926] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 586.130461] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 586.132500] LNet: Accept secure, port 988 [ 587.662668] Key type lgssc registered [ 587.863367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 589.821995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 589.830025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.963153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.768359] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 592.877694] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 594.677646] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 594.681219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.794723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.807825] mount.lustre (4054) used greatest stack depth: 9544 bytes left [ 596.076959] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 597.134020] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 600.794243] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:102 to 0x240000400:129 [ 600.813652] Lustre: 4061:0:(osd_internal.h:1385:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 600.816980] Lustre: 4061:0:(osd_handler.c:1972:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 600.820282] Lustre: 4061:0:(osd_handler.c:1979:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 600.823433] Lustre: 4061:0:(osd_handler.c:1989:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 600.826786] Lustre: 4061:0:(osd_handler.c:1996:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 600.829225] Lustre: 4061:0:(osd_handler.c:2003:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 604.913650] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 606.177157] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 610.812125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.812131] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 610.820780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 613.699543] Lustre: server umount lustre-OST0000 complete [ 614.830853] Lustre: server umount lustre-MDT0000 complete [ 617.891039] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 618.406405] Key type lgssc unregistered [ 618.480996] LNet: 5711:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.484442] LNet: Removed LNI 192.168.204.114@tcp [ 627.399871] alg: No test for adler32 (adler32-zlib) [ 628.153649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.218774] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 628.268567] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 628.270648] LNet: Accept secure, port 988 [ 629.801715] Key type lgssc registered [ 630.033420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.045423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.052402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.202789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.010637] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 635.137144] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.933608] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 636.938121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.036638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.390627] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 639.519339] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 643.037261] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:140 to 0x240000400:161 [ 643.053910] Lustre: 8805:0:(osd_internal.h:1385:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 643.057874] Lustre: 8805:0:(osd_handler.c:1972:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 643.061089] Lustre: 8805:0:(osd_handler.c:1979:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 643.064076] Lustre: 8805:0:(osd_handler.c:1989:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 643.066907] Lustre: 8805:0:(osd_handler.c:1996:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 643.069689] Lustre: 8805:0:(osd_handler.c:2003:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 647.315314] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 653.052246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.052257] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 653.059607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.238370] Lustre: server umount lustre-OST0000 complete [ 656.249339] Lustre: server umount lustre-MDT0000 complete [ 659.142589] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 659.654703] Key type lgssc unregistered [ 659.722928] LNet: 10385:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.726044] LNet: Removed LNI 192.168.204.114@tcp [ 668.856751] alg: No test for adler32 (adler32-zlib) [ 669.608599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.675072] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 669.726427] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 669.728146] LNet: Accept secure, port 988 [ 671.256657] Key type lgssc registered [ 671.481829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.862604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 675.734057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 677.333450] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 677.337271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 678.927886] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 678.931990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 680.747341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 680.761112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 680.767362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.839210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 681.847576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 681.881912] Lustre: lustre-MDT0000: new disk, initializing [ 681.900729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.906026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 682.691386] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 686.238583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 686.260888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.286433] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 686.300222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 686.303042] Lustre: Skipped 1 previous similar message [ 686.337027] Lustre: lustre-MDT0001: new disk, initializing [ 686.359038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 686.370526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 686.373891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 687.126712] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 689.417753] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 690.038750] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 691.829436] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 691.832528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 691.847535] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 691.851253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 691.929149] Lustre: lustre-OST0000: new disk, initializing [ 691.932221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 691.955364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.252880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 695.619911] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 696.473721] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 697.051722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 697.055150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 697.071882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 697.544230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 698.444481] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 698.505052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 702.076290] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 702.076597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.076966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 702.086792] LustreError: Skipped 1 previous similar message [ 705.103537] Lustre: server umount lustre-OST0000 complete [ 706.380229] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 706.383177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.387539] Lustre: Skipped 1 previous similar message [ 706.389631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.391799] Lustre: Skipped 1 previous similar message [ 712.092039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.094427] Lustre: Skipped 3 previous similar messages [ 712.150529] Lustre: server umount lustre-MDT0000 complete [ 713.169376] LustreError: 17681:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092015 with bad export cookie 9991737490400284752 [ 713.170384] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.178662] LustreError: 17681:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 713.294904] Lustre: server umount lustre-MDT0001 complete [ 716.846079] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 03:20:18 (1711092018) [ 718.585396] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 721.588969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 727.726103] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 729.268312] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 731.430157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 733.101084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 734.618295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 734.621869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 736.170843] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 736.174458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.897958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.916624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.984663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 737.993389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 738.029003] Lustre: lustre-MDT0000: new disk, initializing [ 738.046575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.051784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 738.834277] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 742.253150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 742.273213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.298034] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 742.310662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 742.313559] Lustre: Skipped 1 previous similar message [ 742.346868] Lustre: lustre-MDT0001: new disk, initializing [ 742.369305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 742.379341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 742.381976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 743.127287] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 745.417258] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.976820] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 747.736356] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 747.739398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 747.754078] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 747.757515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.109415] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 751.381739] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 752.187686] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 755.184518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 755.187738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 755.204172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 755.283516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 756.056567] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 756.102032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 760.204143] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 760.204650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.204651] Lustre: Skipped 1 previous similar message [ 760.205005] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.215190] LustreError: Skipped 1 previous similar message [ 762.635141] Lustre: server umount lustre-OST0000 complete [ 765.212265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.212648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.212650] Lustre: Skipped 1 previous similar message [ 765.221259] Lustre: Skipped 2 previous similar messages [ 769.678225] Lustre: server umount lustre-MDT0000 complete [ 770.220089] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.225491] LustreError: Skipped 1 previous similar message [ 770.651413] LustreError: 25539:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092073 with bad export cookie 9991737490400285592 [ 770.652536] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.657583] LustreError: 25539:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 770.777710] Lustre: server umount lustre-MDT0001 complete [ 774.342915] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 03:21:16 (1711092076) [ 776.052050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.201955] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.220680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 776.222894] Lustre: Skipped 1 previous similar message [ 776.988175] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 779.227608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.123913] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 781.162891] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 781.748539] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.543259] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 783.547077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 784.913927] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 785.978645] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 786.949478] Lustre: 31331:0:(genops.c:1659:obd_export_evict_by_uuid()) MGS: evicting f9b565fb-2cb8-47cc-9efc-a1e7178c244f at adminstrative request [ 788.640300] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.644883] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 798.645363] Lustre: MGS: Client f9b565fb-2cb8-47cc-9efc-a1e7178c244f (at 0@lo) reconnecting [ 798.648480] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0x8aa9c84f1c4316e0 to 0x8aa9c84f1c431956 [ 798.652175] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 818.760156] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 818.764617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 818.785037] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 818.789410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.834435] Lustre: lustre-OST0001: new disk, initializing [ 818.835989] Lustre: Skipped 1 previous similar message [ 818.837704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 818.839695] Lustre: Skipped 1 previous similar message [ 818.857530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 818.859493] Lustre: Skipped 2 previous similar messages [ 820.046653] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 822.354130] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 823.096477] Lustre: server umount lustre-OST0001 complete [ 825.708309] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.710973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.715717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.717962] Lustre: Skipped 1 previous similar message [ 826.015704] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.177819] Lustre: server umount lustre-OST0000 complete [ 834.396213] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 834.398386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.401964] Lustre: Skipped 1 previous similar message [ 837.218769] Lustre: server umount lustre-MDT0000 complete [ 838.181609] LustreError: 763:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092140 with bad export cookie 9991737490400287062 [ 838.182873] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.191820] LustreError: 763:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 838.309365] Lustre: server umount lustre-MDT0001 complete [ 841.828034] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 03:22:23 (1711092143) [ 842.187769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 844.123581] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 03:22:26 (1711092146) [ 844.461629] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 846.347980] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 03:22:28 (1711092148) [ 846.719809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 848.659854] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 03:22:30 (1711092150) [ 849.037684] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 850.219251] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 850.223995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.485152] loop: module loaded [ 850.489029] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 850.495108] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 850.857842] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 851.361202] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 852.277343] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 854.474530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.475158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 856.441793] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 856.446439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.465722] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 857.470785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 859.150727] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 859.153822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 859.170371] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 859.173856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.178753] LustreError: 5943:0:(obd_config.c:797:class_setup()) setup lustre-OST0000-osd failed (-28) [ 859.181136] LustreError: 5943:0:(obd_mount.c:214:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 859.184313] LustreError: 5943:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 859.188888] LustreError: 5943:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 863.710295] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 865.270673] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 867.385903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 868.966243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 870.462380] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 870.466486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 872.003186] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 872.007857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 873.732030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 873.753342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.823977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 873.831818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 873.866361] Lustre: lustre-MDT0000: new disk, initializing [ 873.884090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.889827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 874.629150] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 878.013017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 878.032549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.128851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 878.873311] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 881.138067] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 881.703918] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 883.421526] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 883.425079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 883.442295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 883.445430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 884.755303] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 887.030704] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 887.806877] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 891.507385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 891.523039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 891.909842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 892.689799] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 892.736194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 896.524648] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 896.524650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.524655] Lustre: Skipped 1 previous similar message [ 896.524958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 896.524959] Lustre: Skipped 5 previous similar messages [ 896.534537] LustreError: Skipped 1 previous similar message [ 899.282409] Lustre: server umount lustre-OST0000 complete [ 906.280158] Lustre: server umount lustre-MDT0000 complete [ 906.540006] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.546104] LustreError: Skipped 2 previous similar messages [ 907.239240] LustreError: 9446:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092209 with bad export cookie 9991737490400287601 [ 907.240888] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.247199] LustreError: 9446:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.343906] Lustre: server umount lustre-MDT0001 complete [ 910.828727] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 03:23:32 (1711092212) [ 912.550976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.675764] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.459577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 915.608359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.480176] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 917.481400] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 918.009614] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 919.666861] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 919.669963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.949946] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 922.010815] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 922.825818] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 922.875148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 923.648238] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 926.737847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 928.375562] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 928.381082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 928.397293] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 928.399863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.435088] Lustre: lustre-OST0001: new disk, initializing [ 928.436736] Lustre: Skipped 2 previous similar messages [ 928.438989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 928.441825] Lustre: Skipped 3 previous similar messages [ 929.650173] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 931.909822] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 932.710717] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 936.350962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 936.353743] Lustre: Skipped 2 previous similar messages [ 936.355264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 936.363317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 936.818393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 937.574702] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 937.623789] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 946.382991] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 950.865733] Lustre: Failing over lustre-MDT0000 [ 950.977913] Lustre: server umount lustre-MDT0000 complete [ 951.388109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.388802] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.395521] Lustre: Skipped 5 previous similar messages [ 955.740065] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.431313] LDISKFS-fs (dm-0): recovery complete [ 964.432758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.461077] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.530265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 964.532291] Lustre: Skipped 6 previous similar messages [ 964.543361] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 965.269606] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 968.854190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 968.856825] Lustre: 18923:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 969.532375] Lustre: 14638:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 969.535939] Lustre: 14638:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 1 previous similar message [ 969.539628] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 969.550751] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 969.852623] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 970.257153] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 970.259798] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 970.562598] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 970.564808] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 970.567569] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 970.570769] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 7 previous similar messages [ 971.375315] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 971.379859] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 971.683611] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 971.685867] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 972.891236] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 972.894730] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 5 previous similar messages [ 973.698600] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 973.701042] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 4 previous similar messages [ 973.703842] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 973.706380] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 5 previous similar messages [ 977.215577] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 977.220249] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 10 previous similar messages [ 977.922631] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 977.925113] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 977.929435] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 977.931959] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 985.346300] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 985.349043] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 20 previous similar messages [ 986.152625] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 986.154659] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 20 previous similar messages [ 986.157292] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 986.160183] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 20 previous similar messages [ 1001.699032] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1001.702516] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 42 previous similar messages [ 1002.405631] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1002.407853] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 1002.411983] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1002.414405] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 1028.774656] Lustre: 21246:0:(ldlm_lib.c:2300:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1028.776774] Lustre: 21246:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1028.778703] Lustre: 21246:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0c0427e8-e5ed-461a-9eae-ce8dcb3e0674@192.168.204.14@tcp [ 1028.782057] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1028.783983] Lustre: 21246:0:(ldlm_lib.c:1824:abort_req_replay_queue()) @@@ aborted: req@ffff880087ed3100 x1794209970418752/t0(8589934752) o35->0c0427e8-e5ed-461a-9eae-ce8dcb3e0674@192.168.204.14@tcp:356/0 lens 392/0 e 0 to 0 dl 1711092341 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1028.794239] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1028.799430] Lustre: lustre-MDT0000: Denying connection for new client 0c0427e8-e5ed-461a-9eae-ce8dcb3e0674 (at 192.168.204.14@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1028.801858] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1028.808019] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1028.808069] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1028.813178] Lustre: Skipped 2 previous similar messages [ 1028.821374] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:73 to 0x280000401:577 [ 1028.824626] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:545 [ 1033.948149] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.950406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.953586] Lustre: Skipped 1 previous similar message [ 1033.955548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.958945] Lustre: Skipped 3 previous similar messages [ 1037.878776] Lustre: server umount lustre-OST0000 complete [ 1039.644197] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.647751] LustreError: Skipped 17 previous similar messages [ 1044.872020] Lustre: server umount lustre-OST0001 complete [ 1051.886357] Lustre: server umount lustre-MDT0000 complete [ 1052.830189] LustreError: 18041:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092355 with bad export cookie 9991737490400338715 [ 1052.831338] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1052.836747] LustreError: 18041:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1052.939517] Lustre: server umount lustre-MDT0001 complete [ 1056.393238] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 03:25:58 (1711092358) [ 1058.251607] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1058.255234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.258969] Lustre: *** cfs_fail_loc=197, val=0*** [ 1075.316930] LustreError: 24184:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 1091.319970] LustreError: 24184:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 1217.328713] Lustre: *** cfs_fail_loc=197, val=0*** [ 1217.330329] Lustre: Skipped 65 previous similar messages [ 1240.338696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.457077] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1242.468784] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1243.037534] Lustre: server umount lustre-OST0000 complete [ 1246.580631] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 03:29:08 (1711092548) [ 1259.300990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.420783] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.425407] LustreError: Skipped 2 previous similar messages [ 1260.163367] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1262.314961] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.141699] LustreError: 26563:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1264.146436] LustreError: 26563:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1264.150591] LustreError: 26563:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1264.250604] Lustre: server umount lustre-MDT0000 complete [ 1267.958234] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1267.962350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.879190] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1273.380269] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1275.469333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1276.951159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1278.368775] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1278.372201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1279.830576] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1279.833543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1281.463284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1281.479481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.545614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1281.547325] Lustre: Skipped 1 previous similar message [ 1281.553632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1281.585396] Lustre: lustre-MDT0000: new disk, initializing [ 1281.606349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1282.340446] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1285.682422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1285.699227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.784018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1286.480805] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1288.734317] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.306434] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1290.945847] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1290.948787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1290.963612] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1290.966581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.216336] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1294.439334] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1295.162130] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1295.748711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1296.219057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1296.956039] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1297.004446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1300.748374] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.748428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.748429] Lustre: Skipped 5 previous similar messages [ 1300.748825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.748827] Lustre: Skipped 7 previous similar messages [ 1300.756908] LustreError: Skipped 3 previous similar messages [ 1303.562141] Lustre: server umount lustre-OST0000 complete [ 1310.554745] Lustre: server umount lustre-MDT0000 complete [ 1311.463371] LustreError: 31505:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092613 with bad export cookie 9991737490400349082 [ 1311.464585] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.471924] LustreError: 31505:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1311.584563] Lustre: server umount lustre-MDT0001 complete [ 1315.069897] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 03:30:17 (1711092617) [ 1317.459053] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1317.941674] Key type lgssc unregistered [ 1318.010873] LNet: 4129:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.013076] LNet: Removed LNI 192.168.204.114@tcp [ 1326.081355] alg: No test for adler32 (adler32-zlib) [ 1326.831639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.887870] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1326.931208] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1326.933243] LNet: Accept secure, port 988 [ 1328.459647] Key type lgssc registered [ 1328.642162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.640434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1332.228484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1333.657651] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1333.660424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1335.177951] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1335.180655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.470395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1338.067302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1338.080435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1338.084332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.148033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1339.155456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1339.187024] Lustre: lustre-MDT0000: new disk, initializing [ 1339.203197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.208473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1339.979953] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1343.321189] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1343.325264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1343.341020] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1343.343651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.411080] Lustre: lustre-OST0000: new disk, initializing [ 1343.413435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1343.416419] Lustre: Skipped 1 previous similar message [ 1343.431997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.638880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1345.641085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1345.645450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1345.652571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1347.975199] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1347.978552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1347.996316] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1347.999516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.031211] Lustre: lustre-OST0001: new disk, initializing [ 1348.033338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1348.050397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.241804] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1353.052311] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.052602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1353.057413] Lustre: Skipped 1 previous similar message [ 1358.060154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.061776] Lustre: Skipped 2 previous similar messages [ 1358.814879] Lustre: server umount lustre-MDT0000 complete [ 1360.434997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1361.554217] LustreError: 8111:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092664 with bad export cookie 258456941266598956 [ 1361.555149] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.559604] LustreError: 8111:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1361.567459] Lustre: server umount lustre-OST0000 complete [ 1362.533957] Lustre: server umount lustre-OST0001 complete [ 1366.028746] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 03:31:08 (1711092668) [ 1367.820034] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1369.668234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1369.942324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.258168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.513367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1374.719995] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 03:31:16 (1711092676) [ 1377.867514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1378.371529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1378.951430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1379.492059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1379.774819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1380.299321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1384.552751] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1386.167087] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1388.395475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.080725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1391.848691] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1391.852553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.522239] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1393.525096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.184011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.200835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.261601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1395.269386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1395.302275] Lustre: lustre-MDT0000: new disk, initializing [ 1395.319926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.326288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1396.069592] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1399.499500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1399.527004] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.569252] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1399.587136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1399.589911] Lustre: Skipped 1 previous similar message [ 1399.623787] Lustre: lustre-MDT0001: new disk, initializing [ 1399.643670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.652192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1399.655805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1400.638989] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1403.029058] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.549007] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1405.167261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1405.171516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1405.188169] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1405.192083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1406.529448] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1408.310735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1408.313199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1408.332817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1408.744932] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.498658] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1409.543309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1410.261612] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1410.304500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1413.324353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.324456] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.324731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.331006] Lustre: Skipped 1 previous similar message [ 1416.832185] Lustre: server umount lustre-OST0000 complete [ 1418.332154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.336146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.337757] Lustre: Skipped 2 previous similar messages [ 1423.339950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.341512] Lustre: Skipped 1 previous similar message [ 1423.785057] Lustre: server umount lustre-MDT0000 complete [ 1424.748271] LustreError: 20226:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092727 with bad export cookie 258456941266600062 [ 1424.749580] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.754993] LustreError: 20226:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1424.850221] Lustre: server umount lustre-MDT0001 complete [ 1428.460489] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 03:32:10 (1711092730) [ 1430.237068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.365317] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.380340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.382305] Lustre: Skipped 1 previous similar message [ 1431.156479] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1433.637119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.443586] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1435.421737] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1435.989836] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1437.680896] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1437.684727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.023275] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1440.061304] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1445.612104] Lustre: *** cfs_fail_loc=159, val=0*** [ 1461.619891] Lustre: lustre-MDT0000: Client 135f10ae-ad2e-4af1-a7ad-6ae95505bbce (at 192.168.204.14@tcp) reconnecting [ 1461.904982] Lustre: *** cfs_fail_loc=159, val=0*** [ 1461.906427] Lustre: Skipped 6 previous similar messages [ 1477.911160] Lustre: lustre-MDT0000: Client 135f10ae-ad2e-4af1-a7ad-6ae95505bbce (at 192.168.204.14@tcp) reconnecting [ 1478.844185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.844905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.851705] Lustre: Skipped 2 previous similar messages [ 1478.854209] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.857168] LustreError: Skipped 1 previous similar message [ 1484.583327] Lustre: server umount lustre-OST0000 complete [ 1488.812248] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1488.816184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.578869] Lustre: server umount lustre-MDT0000 complete [ 1492.611288] LustreError: 24378:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092795 with bad export cookie 258456941266600902 [ 1492.612307] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.617694] LustreError: 24378:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1492.714645] Lustre: server umount lustre-MDT0001 complete [ 1495.770955] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1496.234994] Key type lgssc unregistered [ 1496.298823] LNet: 28302:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.303178] LNet: Removed LNI 192.168.204.114@tcp [ 1504.557729] alg: No test for adler32 (adler32-zlib) [ 1505.308634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.372730] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1505.418728] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1505.421448] LNet: Accept secure, port 988 [ 1506.947711] Key type lgssc registered [ 1507.146377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1508.975318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1508.980493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.123530] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.134168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.903794] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1513.726401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.857905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.858361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1515.817883] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.490853] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1518.300589] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1518.303489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.408316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.555832] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1520.409136] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:42 to 0x280000401:65 [ 1520.509582] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1529.790666] Lustre: *** cfs_fail_loc=159, val=0*** [ 1545.800026] Lustre: lustre-MDT0000: Client c0929a11-4bbe-40b2-ac6a-c3e78b076880 (at 192.168.204.14@tcp) reconnecting [ 1546.190284] Lustre: *** cfs_fail_loc=159, val=0*** [ 1562.194349] Lustre: lustre-MDT0001: Client c0929a11-4bbe-40b2-ac6a-c3e78b076880 (at 192.168.204.14@tcp) reconnecting [ 1562.566237] Lustre: *** cfs_fail_loc=159, val=0*** [ 1562.568457] Lustre: Skipped 4 previous similar messages [ 1578.574976] Lustre: lustre-MDT0001: Client c0929a11-4bbe-40b2-ac6a-c3e78b076880 (at 192.168.204.14@tcp) reconnecting [ 1579.283150] Lustre: 871:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1579.869939] Lustre: *** cfs_fail_loc=159, val=0*** [ 1579.871620] Lustre: Skipped 4 previous similar messages [ 1595.872200] Lustre: lustre-MDT0000: Client f331808d-d6cd-46db-bf2c-1908bfb3b307 (at 192.168.204.14@tcp) reconnecting [ 1596.120172] Lustre: *** cfs_fail_loc=159, val=0*** [ 1596.122230] Lustre: Skipped 13 previous similar messages [ 1612.125863] Lustre: lustre-MDT0000: Client f331808d-d6cd-46db-bf2c-1908bfb3b307 (at 192.168.204.14@tcp) reconnecting [ 1612.508820] Lustre: 1229:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1614.556921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.557293] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1614.557531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1614.576221] Lustre: Skipped 1 previous similar message [ 1619.564413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.567356] Lustre: Skipped 2 previous similar messages [ 1619.890495] Lustre: server umount lustre-OST0000 complete [ 1624.012959] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1624.019976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.030167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.420594] Lustre: server umount lustre-MDT0000 complete [ 1628.855057] LustreError: 30177:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092931 with bad export cookie 16808348563407899089 [ 1628.857789] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.864379] LustreError: 30177:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1629.016050] Lustre: server umount lustre-MDT0001 complete [ 1632.788485] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1633.573397] Key type lgssc unregistered [ 1633.664184] LNet: 2433:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.668877] LNet: Removed LNI 192.168.204.114@tcp [ 1644.598199] alg: No test for adler32 (adler32-zlib) [ 1645.349603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.403660] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1645.450527] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1645.452362] LNet: Accept secure, port 988 [ 1646.979793] Key type lgssc registered [ 1647.363667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1649.678409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1649.683742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.828477] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.839304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.805346] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1654.900352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.045244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1656.135281] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1657.185360] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.824693] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1659.622361] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1659.626179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1659.715466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.717754] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:85 to 0x280000401:129 [ 1661.210028] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1662.812472] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1664.720405] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 1669.583902] Lustre: 6904:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1670.107558] Lustre: 6969:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1670.513251] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1671.419872] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1671.423240] Lustre: Skipped 1 previous similar message [ 1672.027994] Lustre: 7297:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1674.733214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.740112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.740433] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.748264] Lustre: Skipped 1 previous similar message [ 1679.016305] Lustre: server umount lustre-OST0000 complete [ 1684.748560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.749119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.758935] Lustre: Skipped 2 previous similar messages [ 1686.512339] Lustre: server umount lustre-MDT0000 complete [ 1687.906820] LustreError: 7711:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092990 with bad export cookie 9526787214250758767 [ 1687.909626] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.923453] LustreError: 7711:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1688.042767] Lustre: server umount lustre-MDT0001 complete [ 1691.838641] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1692.630541] Key type lgssc unregistered [ 1692.723153] LNet: 8507:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.728088] LNet: Removed LNI 192.168.204.114@tcp [ 1702.969243] alg: No test for adler32 (adler32-zlib) [ 1703.719604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.786231] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1703.836008] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1703.837801] LNet: Accept secure, port 988 [ 1705.366680] Key type lgssc registered [ 1705.576861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.991174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.996301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.135483] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1709.150484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.876113] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1712.098455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.187849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.003389] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1714.123502] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.849243] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1716.550604] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1716.554103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1716.641232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.996863] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1718.642068] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:85 to 0x280000401:161 [ 1719.153900] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1721.643991] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 1725.879825] Lustre: *** cfs_fail_loc=159, val=0*** [ 1741.889785] Lustre: lustre-MDT0000: Client ff143128-eb7d-4631-9904-329b5d559396 (at 192.168.204.14@tcp) reconnecting [ 1743.692235] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.695346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.700066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.685375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.685675] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.685698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.700919] Lustre: Skipped 1 previous similar message [ 1748.757570] Lustre: server umount lustre-OST0000 complete [ 1751.692331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.692591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.698141] Lustre: Skipped 1 previous similar message [ 1755.856068] Lustre: server umount lustre-MDT0000 complete [ 1756.700074] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.703239] LustreError: Skipped 1 previous similar message [ 1756.822412] LustreError: 10390:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093059 with bad export cookie 12479050407346279092 [ 1756.823978] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.827773] LustreError: 10390:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1756.934975] Lustre: server umount lustre-MDT0001 complete [ 1760.210582] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1760.688786] Key type lgssc unregistered [ 1760.752945] LNet: 14177:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.756133] LNet: Removed LNI 192.168.204.114@tcp [ 1769.220940] alg: No test for adler32 (adler32-zlib) [ 1769.974661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.033601] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1770.086530] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1770.088484] LNet: Accept secure, port 988 [ 1771.621786] Key type lgssc registered [ 1771.812476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1773.967919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1773.978420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.120462] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.131603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.059079] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1778.415624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.573089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1779.660671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1781.121302] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1781.729687] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1783.713311] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1783.717944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.839197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.840261] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:170 to 0x280000401:193 [ 1785.231546] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1786.691360] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1788.840397] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 1804.724081] Lustre: lustre-OST0000: evicting 3e30dd0b-6725-4f85-b8e6-5682de030fcc (at 192.168.204.14@tcp) by administrative request [ 1804.728565] Lustre: Skipped 1 previous similar message [ 1828.908373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.908579] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.909007] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.914752] Lustre: Skipped 1 previous similar message [ 1833.916423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.917941] Lustre: Skipped 2 previous similar messages [ 1834.611409] Lustre: server umount lustre-OST0000 complete [ 1838.636182] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.638366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.642333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1841.652951] Lustre: server umount lustre-MDT0000 complete [ 1842.703418] LustreError: 16052:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093145 with bad export cookie 12770208352824334498 [ 1842.704602] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.714401] LustreError: 16052:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1842.821106] Lustre: server umount lustre-MDT0001 complete [ 1845.927218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1846.440821] Key type lgssc unregistered [ 1846.511848] LNet: 20368:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.514267] LNet: Removed LNI 192.168.204.114@tcp [ 1875.267345] alg: No test for adler32 (adler32-zlib) [ 1876.017596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.076499] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1876.126523] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1876.128792] LNet: Accept secure, port 988 [ 1877.654707] Key type lgssc registered [ 1877.854655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1879.692864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1879.698324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.835619] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.846258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.599018] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1883.838608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.938379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.684874] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1885.790534] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.510025] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1888.189380] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1888.193150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.287707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.513088] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1890.548339] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.288196] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:195 to 0x280000401:225 [ 1894.288789] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 1899.293229] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1899.293472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.294057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.871741] Lustre: server umount lustre-OST0000 complete [ 1903.948048] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.950725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.954756] Lustre: Skipped 1 previous similar message [ 1903.956560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1903.958681] Lustre: Skipped 1 previous similar message [ 1907.928654] Lustre: server umount lustre-MDT0000 complete [ 1908.843407] LustreError: 28852:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093211 with bad export cookie 5503862414310465120 [ 1908.844732] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.850830] LustreError: 28852:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1908.948500] Lustre: server umount lustre-MDT0001 complete [ 1911.641706] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1912.072715] Key type lgssc unregistered [ 1912.134805] LNet: 29647:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.137072] LNet: Removed LNI 192.168.204.114@tcp [ 1919.667588] alg: No test for adler32 (adler32-zlib) [ 1920.418592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.475759] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1920.521800] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1920.523207] LNet: Accept secure, port 988 [ 1922.048676] Key type lgssc registered [ 1922.225439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.786253] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1923.789165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1926.984792] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 03:40:29 (1711093229) [ 1930.551565] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1931.974913] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1934.094596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.625660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1937.023730] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1937.028006] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1938.496860] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1938.500791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1940.134402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1940.151534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1940.155326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.226793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1941.234407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1941.267731] Lustre: lustre-MDT0000: new disk, initializing [ 1941.284276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.289144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1941.980154] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1945.301793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1945.320023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.346870] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1945.359943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1945.362512] Lustre: Skipped 1 previous similar message [ 1945.399201] Lustre: lustre-MDT0001: new disk, initializing [ 1945.421671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1945.436863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1945.440012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1946.125632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1948.321566] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1948.844741] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1950.515095] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1950.518724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1950.537306] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1950.540675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1950.617451] Lustre: lustre-OST0000: new disk, initializing [ 1950.619983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1950.638217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.751210] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1953.920366] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1957.555764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1957.558086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1957.571731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1962.572152] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.572364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.572689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1962.579715] LustreError: Skipped 1 previous similar message [ 1967.475730] Lustre: server umount lustre-OST0000 complete [ 1970.444240] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1970.446273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.449027] Lustre: Skipped 1 previous similar message [ 1970.451034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.453056] Lustre: Skipped 1 previous similar message [ 1972.588155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.588343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.595134] Lustre: Skipped 1 previous similar message [ 1974.496373] Lustre: server umount lustre-MDT0000 complete [ 1975.426391] LustreError: 3954:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093277 with bad export cookie 996362589169131967 [ 1975.427694] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.435429] LustreError: 3954:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1975.544380] Lustre: server umount lustre-MDT0001 complete [ 1978.308650] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1978.791715] Key type lgssc unregistered [ 1978.858075] LNet: 7068:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.860384] LNet: Removed LNI 192.168.204.114@tcp [ 1987.291091] alg: No test for adler32 (adler32-zlib) [ 1988.042569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.095637] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1988.137581] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1988.139329] LNet: Accept secure, port 988 [ 1989.663683] Key type lgssc registered [ 1989.919085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.877007] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1991.881571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.008564] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.017878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.958442] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1996.235693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.357427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.370595] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1998.451451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1999.058692] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.747420] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2000.750466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.848390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.066265] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2002.848467] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:3 to 0x280000401:33 [ 2003.092518] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2009.732769] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2009.738235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.744538] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2009.747552] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2009.750659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2010.407809] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2010.409472] Lustre: Skipped 2 previous similar messages [ 2010.413002] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.416348] Lustre: Skipped 2 previous similar messages [ 2010.417652] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2010.419468] Lustre: Skipped 2 previous similar messages [ 2010.420621] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2010.424974] LustreError: Skipped 2 previous similar messages [ 2010.426728] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2010.429191] Lustre: Skipped 2 previous similar messages [ 2011.678930] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2011.682754] Lustre: Skipped 4 previous similar messages [ 2011.692023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.699759] Lustre: Skipped 4 previous similar messages [ 2011.702352] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2011.705234] Lustre: Skipped 4 previous similar messages [ 2011.707039] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2011.711647] LustreError: Skipped 4 previous similar messages [ 2011.713470] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2011.716128] Lustre: Skipped 4 previous similar messages [ 2013.995327] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2013.998894] Lustre: Skipped 6 previous similar messages [ 2014.005191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.010639] Lustre: Skipped 6 previous similar messages [ 2014.012576] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2014.015860] Lustre: Skipped 6 previous similar messages [ 2014.017811] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2014.022581] LustreError: Skipped 6 previous similar messages [ 2014.025581] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2014.029221] Lustre: Skipped 6 previous similar messages [ 2018.019145] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2018.021842] Lustre: Skipped 17 previous similar messages [ 2018.026883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.029817] Lustre: Skipped 17 previous similar messages [ 2018.031031] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2018.032722] Lustre: Skipped 17 previous similar messages [ 2018.033948] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2018.037005] LustreError: Skipped 17 previous similar messages [ 2018.038480] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2018.040322] Lustre: Skipped 17 previous similar messages [ 2026.085298] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2026.086971] Lustre: Skipped 36 previous similar messages [ 2026.090441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.093148] Lustre: Skipped 36 previous similar messages [ 2026.094372] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2026.096316] Lustre: Skipped 36 previous similar messages [ 2026.097491] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2026.100255] LustreError: Skipped 36 previous similar messages [ 2026.101687] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2026.104419] Lustre: Skipped 36 previous similar messages [ 2036.630397] LustreError: 8985:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2036.633611] LustreError: 8985:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2042.416633] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2042.416957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.416959] Lustre: Skipped 59 previous similar messages [ 2042.417356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.426520] LustreError: Skipped 5 previous similar messages [ 2045.916194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.916686] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.916687] LustreError: Skipped 2 previous similar messages [ 2045.922821] Lustre: Skipped 1 previous similar message [ 2047.569590] Lustre: server umount lustre-OST0000 complete [ 2050.924549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2050.926574] Lustre: Skipped 1 previous similar message [ 2054.597702] Lustre: server umount lustre-MDT0000 complete [ 2055.643446] LustreError: 8949:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093358 with bad export cookie 17759894657738448998 [ 2055.644781] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.651692] LustreError: 8949:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2055.754155] Lustre: server umount lustre-MDT0001 complete [ 2058.644900] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2059.079189] Key type lgssc unregistered [ 2059.138878] LNet: 21774:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.141217] LNet: Removed LNI 192.168.204.114@tcp [ 2066.985754] alg: No test for adler32 (adler32-zlib) [ 2067.736607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2067.789460] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2067.833245] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2067.835325] LNet: Accept secure, port 988 [ 2069.360665] Key type lgssc registered [ 2069.542903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.317684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.322815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.442979] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.452478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.125260] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2075.245942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.345075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.094979] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2077.125735] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.690071] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.529191] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2079.532543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.653771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.017592] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2081.653245] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:1171 to 0x280000401:1217 [ 2082.091698] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2088.448274] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2088.923607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.928194] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2088.930409] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2088.934169] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2088.934188] LustreError: 23687:0:(osp_precreate.c:1282:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2089.660331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.660809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.666651] Lustre: Skipped 1 previous similar message [ 2091.451998] Lustre: server umount lustre-OST0000 complete [ 2109.033629] LustreError: 23687:0:(osp_precreate.c:1282:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2118.651077] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2118.654265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.738523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.737840] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2119.742466] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2119.925482] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2120.941957] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2121.310131] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2121.884413] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012abf8000.ost_server_uuid 50 [ 2122.224554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012abf8000.ost_server_uuid in FULL state after 0 sec [ 2124.748529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.752026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2124.754500] Lustre: Skipped 1 previous similar message [ 2128.942904] Lustre: server umount lustre-OST0000 complete [ 2130.412262] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.415211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.419693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.763990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.764247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.769285] Lustre: Skipped 1 previous similar message [ 2135.944809] Lustre: server umount lustre-MDT0000 complete [ 2136.890241] LustreError: 27836:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093439 with bad export cookie 13113745336134562201 [ 2136.891130] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.895738] LustreError: 27836:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2136.994835] Lustre: server umount lustre-MDT0001 complete [ 2139.724732] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2140.178721] Key type lgssc unregistered [ 2140.242902] LNet: 29014:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.246055] LNet: Removed LNI 192.168.204.114@tcp [ 2149.398998] alg: No test for adler32 (adler32-zlib) [ 2150.149637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.211459] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2150.259180] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2150.260552] LNet: Accept secure, port 988 [ 2151.788742] Key type lgssc registered [ 2152.004628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2154.124736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2154.129648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.253670] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.263631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.029171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2158.118133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.210941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.921670] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2161.127979] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2161.132592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.244082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.529205] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2164.782895] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2164.786392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2164.804113] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2164.807085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2164.839881] Lustre: lustre-OST0001: new disk, initializing [ 2164.842065] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2164.861699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.863227] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:1219 to 0x280000401:1249 [ 2166.205729] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2170.027748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2170.032012] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2170.054376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2171.540491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2177.002706] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2188.252138] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.255022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.260471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.076419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.076968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.084051] Lustre: Skipped 2 previous similar messages [ 2193.808357] Lustre: server umount lustre-MDT0000 complete [ 2194.706249] LustreError: 31302:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093497 with bad export cookie 16342499631108639545 [ 2194.707186] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.711544] LustreError: 31302:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2194.801933] Lustre: server umount lustre-MDT0001 complete [ 2205.730681] Lustre: 3492:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093498/real 1711093498] req@ffff88007a082300 x1794211527561088/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093508 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2205.756599] Lustre: server umount lustre-OST0000 complete [ 2216.884627] Lustre: 3766:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093509/real 1711093509] req@ffff88007a082300 x1794211527561472/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093519 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2216.965538] Lustre: server umount lustre-OST0001 complete [ 2217.808745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2217.834213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2218.121167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2218.145141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2218.440400] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2218.444009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2218.461363] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2218.465612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2218.753328] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2218.757057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2218.780181] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2218.784258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2221.768182] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2223.969062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.108913] LustreError: 137-5: mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.130745] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.852134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2226.890025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.781406] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2230.081995] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2230.086308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.398817] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2233.625969] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2233.629560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.677138] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.679028] Lustre: Skipped 2 previous similar messages [ 2234.677344] Lustre: mylustre-OST0000: deleting orphan objects from 0x280000401:1251 to 0x280000401:1281 [ 2234.680975] Lustre: mylustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:33 [ 2234.909458] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2241.155826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2242.081646] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2251.708637] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.709066] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.709068] Lustre: Skipped 3 previous similar messages [ 2251.720950] Lustre: Skipped 3 previous similar messages [ 2256.716957] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.719548] Lustre: Skipped 6 previous similar messages [ 2256.899214] Lustre: server umount mylustre-MDT0000 complete [ 2257.911803] LustreError: 5441:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093560 with bad export cookie 16342499631108642247 [ 2257.913640] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.921301] LustreError: 5441:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2258.037203] Lustre: server umount mylustre-MDT0001 complete [ 2269.133651] Lustre: 9957:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093561/real 1711093561] req@ffff88013120dc00 x1794211527584640/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093571 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2269.155671] Lustre: server umount mylustre-OST0000 complete [ 2280.150679] Lustre: 10230:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093572/real 1711093572] req@ffff88013120dc00 x1794211527585152/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093582 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2280.232681] Lustre: server umount mylustre-OST0001 complete [ 2281.073946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2281.094638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.377038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2281.391904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.667677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2281.671322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2281.692834] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2281.696747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.977837] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2281.981683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2282.003289] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2282.007590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2285.012239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2287.421475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.562538] LustreError: 137-5: tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.577590] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.336903] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2290.486167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.344054] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2293.883571] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2293.889292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.981111] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2298.024076] LustreError: 137-5: tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.042231] Lustre: tfs-OST0000: deleting orphan objects from 0x280000401:1283 to 0x280000401:1313 [ 2301.406818] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2301.424220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.556509] hrtimer: interrupt took 4265044 ns [ 2303.247151] Lustre: tfs-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:65 [ 2304.646042] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2309.768256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2312.479063] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2325.646847] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2325.649999] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.655449] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.738280] Lustre: server umount tfs-MDT0000 complete [ 2331.316220] LustreError: 12788:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093633 with bad export cookie 16342499631108644102 [ 2331.317638] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2331.334249] LustreError: 12788:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.633071] Lustre: server umount tfs-MDT0001 complete [ 2343.940431] Lustre: 16325:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093636/real 1711093636] req@ffff880072d29880 x1794211527605696/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093646 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2344.016908] Lustre: server umount tfs-OST0000 complete [ 2356.202650] Lustre: 16599:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093648/real 1711093648] req@ffff880072d2a680 x1794211527606080/t0(0) o39->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093658 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2356.322666] Lustre: server umount tfs-OST0001 complete [ 2358.046740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2358.099345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.659934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2358.723635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.217686] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2359.224564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2359.265273] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2359.273150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.729596] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2359.739279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2359.773863] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2359.781258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2365.616586] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2370.116154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.482337] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.488069] LustreError: Skipped 2 previous similar messages [ 2370.518488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.522024] Lustre: Skipped 3 previous similar messages [ 2372.140140] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2377.151296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.874995] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2383.974462] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2383.992700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.206328] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.223094] LustreError: Skipped 1 previous similar message [ 2387.249169] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2392.514632] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2392.522234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.476576] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2395.873022] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:1283 to 0x280000401:1345 [ 2395.873307] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:97 [ 2399.765468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2401.633317] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2412.401099] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 03:48:34 (1711093714) [ 2425.933092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.936597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.936599] Lustre: Skipped 4 previous similar messages [ 2425.945793] Lustre: Skipped 5 previous similar messages [ 2429.663548] Lustre: server umount lustre-MDT0000 complete [ 2430.806720] LustreError: 18263:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093733 with bad export cookie 16342499631108645733 [ 2430.808440] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.817342] LustreError: 18263:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.940076] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.943566] LustreError: Skipped 2 previous similar messages [ 2437.069412] Lustre: server umount lustre-MDT0001 complete [ 2445.171895] Lustre: server umount lustre-OST0000 complete [ 2453.161313] Lustre: server umount lustre-OST0001 complete [ 2456.064641] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2460.134956] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2465.226767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2468.924214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2472.282244] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2472.309326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2475.927523] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2475.938344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.348172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.397021] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2479.415302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2479.563979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2479.581926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2479.619375] Lustre: lustre-MDT0000: new disk, initializing [ 2479.670671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.673439] Lustre: Skipped 3 previous similar messages [ 2479.679948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2481.458218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2487.120211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2487.147398] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2487.156178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2487.226099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2487.229098] Lustre: Skipped 1 previous similar message [ 2487.271144] Lustre: lustre-MDT0001: new disk, initializing [ 2487.343934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2487.350638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2488.613332] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2493.200534] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2493.209277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2493.239069] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2493.243286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.343180] Lustre: lustre-OST0000: new disk, initializing [ 2493.347545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2494.485111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2494.488457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2494.526542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2495.205982] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2500.712968] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2500.719667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2500.759763] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2500.766890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.850382] Lustre: lustre-OST0001: new disk, initializing [ 2500.855177] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2502.336120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2502.338720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2502.378081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2502.866521] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2510.880742] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 03:50:12 (1711093812) [ 2518.792393] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 03:50:20 (1711093820) [ 2532.417087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.423101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2532.423102] Lustre: Skipped 7 previous similar messages [ 2532.443889] Lustre: Skipped 4 previous similar messages [ 2536.684389] Lustre: server umount lustre-OST0000 complete [ 2537.425960] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.441307] LustreError: Skipped 3 previous similar messages [ 2542.413254] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2545.149451] Lustre: server umount lustre-MDT0000 complete [ 2547.421652] LustreError: 27119:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093849 with bad export cookie 16342499631108647252 [ 2547.423546] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.438321] LustreError: 27119:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2553.601235] Lustre: server umount lustre-MDT0001 complete [ 2559.009277] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2571.242179] Lustre: server umount lustre-OST0001 complete [ 2574.390122] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2581.869551] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2586.710139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2589.959196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2592.977754] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2592.987992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2596.567034] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2596.575009] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2600.169202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2600.211569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.330269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2600.333337] Lustre: Skipped 1 previous similar message [ 2600.357319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2600.398919] Lustre: lustre-MDT0000: new disk, initializing [ 2600.451417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2602.162481] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2608.367426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2608.404985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.625637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.627783] Lustre: Skipped 4 previous similar messages [ 2608.671666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2610.354978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2614.293883] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.647331] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2619.510865] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2619.529494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.565797] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2619.574029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2619.722820] Lustre: lustre-OST0000: new disk, initializing [ 2619.726636] Lustre: Skipped 1 previous similar message [ 2619.729028] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2619.733337] Lustre: Skipped 2 previous similar messages [ 2621.602784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2621.606445] Lustre: Skipped 1 previous similar message [ 2621.609094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2621.700452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2622.546834] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2626.411665] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.647265] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.658620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.678407] Lustre: Skipped 5 previous similar messages [ 2631.681162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.681164] Lustre: Skipped 6 previous similar messages [ 2635.991779] Lustre: server umount lustre-OST0000 complete [ 2638.655559] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2638.658690] LustreError: Skipped 1 previous similar message [ 2644.287726] Lustre: server umount lustre-MDT0000 complete [ 2646.340334] LustreError: 3017:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093948 with bad export cookie 16342499631108648680 [ 2646.351342] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.370032] LustreError: 3017:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2646.724460] Lustre: server umount lustre-MDT0001 complete [ 2651.922993] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2653.055140] Key type lgssc unregistered [ 2653.155436] LNet: 6910:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2653.160746] LNet: Removed LNI 192.168.204.114@tcp [ 2669.780734] alg: No test for adler32 (adler32-zlib) [ 2670.558738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2670.746477] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2670.852781] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2670.856551] LNet: Accept secure, port 988 [ 2672.452239] Key type lgssc registered [ 2673.025321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.336203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2677.351711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.694567] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.725816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.671633] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2685.786490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.978815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.933727] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2690.499894] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2691.750448] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2696.150635] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2696.161697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.429160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.087186] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2701.570053] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2702.163745] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2703.610115] Lustre: server umount lustre-OST0000 complete [ 2709.456031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.465392] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.495560] Lustre: Skipped 1 previous similar message [ 2710.267393] Lustre: server umount lustre-MDT0000 complete [ 2710.501235] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.512867] LustreError: Skipped 2 previous similar messages [ 2712.582101] LustreError: 8862:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094015 with bad export cookie 5669781906564607568 [ 2712.585955] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.603281] LustreError: 8862:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2712.788472] Lustre: server umount lustre-MDT0001 complete [ 2718.413433] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2720.418806] Key type lgssc unregistered [ 2720.576455] LNet: 12569:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2720.591898] LNet: Removed LNI 192.168.204.114@tcp [ 2730.895778] alg: No test for adler32 (adler32-zlib) [ 2731.746817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.916958] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2732.097785] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2732.106217] LNet: Accept secure, port 988 [ 2733.759788] Key type lgssc registered [ 2734.593838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.150450] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.165029] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2741.179639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.395159] LustreError: 13798:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 2794.424841] LustreError: 13798:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 2944.537944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.517350] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2950.578438] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2952.095865] Lustre: server umount lustre-OST0000 complete [ 2962.986019] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2965.150034] Key type lgssc unregistered [ 2965.402157] LNet: 15628:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2965.405668] LNet: Removed LNI 192.168.204.114@tcp [ 2998.299428] alg: No test for adler32 (adler32-zlib) [ 2999.052817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2999.218774] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2999.327664] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2999.331355] LNet: Accept secure, port 988 [ 3000.918603] Key type lgssc registered [ 3001.507814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3008.798922] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.603927] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.364360] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3010.376796] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.065039] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3011.074321] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.693842] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3014.067712] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3016.187393] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3016.199375] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3018.385059] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3018.393268] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3020.708518] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.733083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.743652] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.778901] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3020.783436] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3020.795232] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3020.800390] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3020.809153] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3020.815606] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3020.826426] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3020.832210] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3020.840843] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3020.849941] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3020.856335] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3020.861530] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3020.871346] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3020.879115] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3024.207893] Lustre: server umount MGS complete [ 3024.653087] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.876104] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.908565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.941031] Lustre: lustre-MDT0000: Aborting client recovery [ 3025.434305] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.477400] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.501348] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3025.506565] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3025.514806] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3025.523807] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3025.531366] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3025.538719] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3025.545674] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3025.559447] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3025.658085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3025.687856] Lustre: lustre-MDT0001: Aborting client recovery [ 3026.165915] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 3026.173155] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.204986] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 3026.210376] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.226954] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3026.234315] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3026.240958] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3026.251517] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3026.259682] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3026.266636] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3026.275455] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3026.345608] Lustre: Found index 0 for lustre-OST0000, updating log [ 3026.403231] Lustre: lustre-OST0000: Aborting recovery [ 3026.863667] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 3026.870054] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.897488] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 3026.904192] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.932707] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3026.937200] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3026.944475] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3026.950198] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3026.960436] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3026.968367] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3026.974568] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3027.021802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3027.026781] Lustre: Skipped 1 previous similar message [ 3027.038233] Lustre: lustre-OST0001: Aborting recovery [ 3032.043448] LustreError: 21037:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3035.036393] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 3035.039796] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 3035.050500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3035.060548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3036.040353] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 3036.043469] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 3037.906317] LustreError: 21007:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 877us: rc = -2 [ 3050.669161] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.678100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.687435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.043642] Lustre: server umount lustre-MDT0000 complete [ 3055.603238] LustreError: 20992:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094358 with bad export cookie 12197495060144566976 [ 3055.603403] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.614115] LustreError: 20992:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3056.077674] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.078135] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.080378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3056.080379] Lustre: Skipped 4 previous similar messages [ 3056.097397] Lustre: Skipped 4 previous similar messages [ 3060.087223] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.087533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3060.087534] Lustre: Skipped 1 previous similar message [ 3060.121393] LustreError: Skipped 2 previous similar messages [ 3061.827804] Lustre: server umount lustre-MDT0001 complete [ 3068.324320] Lustre: server umount lustre-OST0000 complete [ 3074.953562] Lustre: server umount lustre-OST0001 complete [ 3081.539278] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 03:59:43 (1711094383) [ 3094.495332] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3098.151798] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3103.209597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.995665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.233482] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3110.240731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.525355] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3113.538985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.277091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.306278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.428626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3117.443410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3117.488004] Lustre: lustre-MDT0000: new disk, initializing [ 3117.537849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.552039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3119.370652] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3125.554813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3125.591402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.640350] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3125.673785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3125.676478] Lustre: Skipped 1 previous similar message [ 3125.748446] Lustre: lustre-MDT0001: new disk, initializing [ 3125.820974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.845181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3125.850251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3127.574559] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3131.206162] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3132.496431] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3136.115697] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3136.128468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3136.177333] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3136.187547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.323322] Lustre: lustre-OST0000: new disk, initializing [ 3136.326127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3136.371301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.414732] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3140.874609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3140.878866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3140.949157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3142.063696] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3145.353110] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3151.378920] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3156.557030] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3195.999350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.999863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.999865] Lustre: Skipped 1 previous similar message [ 3196.009698] Lustre: Skipped 1 previous similar message [ 3201.030254] Lustre: server umount lustre-OST0000 complete [ 3205.916630] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.919805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.925629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.928078] Lustre: Skipped 1 previous similar message [ 3209.255921] Lustre: server umount lustre-MDT0000 complete [ 3211.021292] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.038222] LustreError: Skipped 1 previous similar message [ 3211.704083] LustreError: 26778:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094514 with bad export cookie 12197495060144569643 [ 3211.708356] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3211.742445] LustreError: 26778:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3211.936830] Lustre: server umount lustre-MDT0001 complete [ 3217.368472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.056262] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3221.952965] Lustre: server umount MGS complete [ 3229.149998] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.404035] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.415122] LustreError: 1221:0:(qmt_pool.c:1518:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3229.458179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.592311] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3237.345452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.775864] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3242.403907] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3243.880056] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.789565] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3247.802658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3250.987648] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3253.437214] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3253.701181] Lustre: MGS: Client f0b92257-2b25-40c7-9e2c-b1c0f0703802 (at 192.168.204.14@tcp) reconnecting [ 3255.432986] Lustre: DEBUG MARKER: conf-sanity test_109a: @@@@@@ FAIL: wrong llite parameter after clear_conf [ 3284.696892] Lustre: DEBUG MARKER: == conf-sanity test 109b: test lctl clear_conf one config ========================================================== 04:03:06 (1711094586) [ 3287.677154] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.683659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.690720] Lustre: Skipped 1 previous similar message [ 3287.693211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.698050] Lustre: Skipped 1 previous similar message [ 3292.078375] Lustre: server umount lustre-MDT0000 complete [ 3293.117876] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.128012] LustreError: Skipped 3 previous similar messages [ 3294.360218] LustreError: 1191:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094596 with bad export cookie 12197495060144571435 [ 3294.363816] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.392774] LustreError: 1191:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3294.612577] Lustre: server umount lustre-MDT0001 complete [ 3307.004647] Lustre: 5590:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711094599/real 1711094599] req@ffff88009d64c700 x1794212417849216/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711094609 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 3307.144536] Lustre: server umount lustre-OST0000 complete [ 3318.300759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3323.352705] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3328.974214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3332.776906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3336.350500] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3336.359544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3340.557255] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3340.576878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3345.083031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3345.142283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.340533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3345.343525] Lustre: Skipped 1 previous similar message [ 3345.363153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3345.422768] Lustre: lustre-MDT0000: new disk, initializing [ 3345.500676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.508479] Lustre: Skipped 2 previous similar messages [ 3345.530874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3347.398011] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3354.664411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.710940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.785501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3354.788751] Lustre: Skipped 1 previous similar message [ 3354.882576] Lustre: lustre-MDT0001: new disk, initializing [ 3354.961817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3354.971050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3356.971938] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3362.077125] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3363.751775] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3368.349613] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3368.363820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.400172] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3368.409857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3368.631731] Lustre: lustre-OST0000: new disk, initializing [ 3368.637548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3372.116879] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3374.695001] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3374.702605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3374.787367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3376.503295] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3380.312052] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3380.314645] Lustre: Skipped 1 previous similar message [ 3429.872997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.873946] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3429.874335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.874337] Lustre: Skipped 3 previous similar messages [ 3429.897219] Lustre: Skipped 3 previous similar messages [ 3432.078261] Lustre: server umount lustre-OST0000 complete [ 3440.588813] Lustre: server umount lustre-MDT0000 complete [ 3442.834935] LustreError: 9260:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094745 with bad export cookie 12197495060144572387 [ 3442.838225] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3442.856382] LustreError: 9260:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3443.098357] Lustre: server umount lustre-MDT0001 complete [ 3448.338533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.190995] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3453.221829] Lustre: server umount MGS complete [ 3460.562593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.888238] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.914980] LustreError: 15878:0:(qmt_pool.c:1518:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3460.957256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.964387] Lustre: Skipped 2 previous similar messages [ 3463.036150] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3468.542953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.971577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3473.536823] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3475.000083] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3478.853726] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3478.862647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.742292] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3484.277547] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3504.141777] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.147054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.155429] Lustre: Skipped 2 previous similar messages [ 3504.157664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.161770] Lustre: Skipped 5 previous similar messages [ 3509.991827] Lustre: server umount lustre-OST0000 complete [ 3513.900349] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.906959] LustreError: Skipped 1 previous similar message [ 3517.753177] Lustre: server umount lustre-MDT0000 complete [ 3519.299159] LustreError: 17723:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094821 with bad export cookie 12197495060144574186 [ 3519.300383] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3519.311791] LustreError: 17723:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3519.468602] Lustre: server umount lustre-MDT0001 complete [ 3523.630159] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3524.550317] Key type lgssc unregistered [ 3524.654982] LNet: 20128:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3524.659433] LNet: Removed LNI 192.168.204.114@tcp [ 3538.794365] alg: No test for adler32 (adler32-zlib) [ 3539.546279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3539.632457] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 3539.705742] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3539.708974] LNet: Accept secure, port 988 [ 3541.256067] Key type lgssc registered [ 3541.602880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3544.766397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3544.774191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.991362] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3546.014139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.149447] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3550.574771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.811824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3552.303006] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3554.117371] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3555.320760] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3558.985674] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3559.000459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3559.231785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.034093] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3564.254778] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3567.646857] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3567.653809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3567.684639] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3567.691968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.755191] Lustre: lustre-OST0001: new disk, initializing [ 3567.757942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3567.810488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3570.112235] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3572.825799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3572.830618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3572.856619] LustreError: 25686:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3573.867844] LustreError: 25686:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3573.873533] LustreError: 25686:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3574.882383] LustreError: 25686:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3574.890989] LustreError: 25686:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 3576.017450] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3576.109422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3576.892052] LustreError: 25692:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3576.897247] LustreError: 25692:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 3577.313444] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b639f800.ost_server_uuid 50 [ 3578.124108] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b639f800.ost_server_uuid in FULL state after 0 sec [ 3579.353581] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b639f800.ost_server_uuid 50 [ 3580.119373] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800b639f800.ost_server_uuid in FULL state after 0 sec [ 3580.943828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3592.912617] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.914524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3592.929113] Lustre: Skipped 1 previous similar message [ 3594.252343] Lustre: lustre-OST0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 3594.255561] Lustre: Skipped 1 previous similar message [ 3597.916794] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3597.923532] Lustre: Skipped 1 previous similar message [ 3598.845234] Lustre: server umount lustre-OST0001 complete [ 3599.210632] LustreError: 137-5: lustre-OST0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.928626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.931396] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.944634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3602.947791] Lustre: Skipped 1 previous similar message [ 3604.091135] Lustre: server umount lustre-OST0000 complete [ 3607.935536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.943243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.943245] Lustre: Skipped 1 previous similar message [ 3607.956225] Lustre: Skipped 1 previous similar message [ 3612.134945] Lustre: server umount lustre-MDT0000 complete [ 3612.940554] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.945751] LustreError: Skipped 2 previous similar messages [ 3614.553117] LustreError: 25057:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094917 with bad export cookie 8816531442503288178 [ 3614.559878] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.890697] Lustre: server umount lustre-MDT0001 complete [ 3620.899222] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3621.891082] Key type lgssc unregistered [ 3622.018211] LNet: 27835:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3622.023115] LNet: Removed LNI 192.168.204.114@tcp [ 3635.602411] alg: No test for adler32 (adler32-zlib) [ 3636.358099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3636.479783] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 3636.555411] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3636.558046] LNet: Accept secure, port 988 [ 3638.112680] Key type lgssc registered [ 3638.544159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3642.074788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3642.087275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.304918] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.338723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.697949] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3646.881074] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3650.573576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.811233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3652.370345] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3657.146787] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3657.152772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.298975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.242116] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3661.002636] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3661.298392] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.718366] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3663.726200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.852464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3666.292797] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3667.886460] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:68 to 0x280000401:97 [ 3667.894587] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:33 [ 3668.235672] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3670.943170] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3671.018830] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3763.023624] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.031579] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3764.278363] Lustre: server umount lustre-OST0001 complete [ 3768.028786] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.028883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.028885] Lustre: Skipped 1 previous similar message [ 3768.030361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.030362] Lustre: Skipped 1 previous similar message [ 3768.062301] LustreError: Skipped 1 previous similar message [ 3772.827421] Lustre: server umount lustre-OST0000 complete [ 3775.966725] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3775.972495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.983771] Lustre: Skipped 1 previous similar message [ 3775.991387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3775.993996] Lustre: Skipped 1 previous similar message [ 3778.044310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3778.044680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.503596] Lustre: server umount lustre-MDT0000 complete [ 3782.573843] LustreError: 29722:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711095085 with bad export cookie 9107361227827126722 [ 3782.576977] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3782.584493] LustreError: 29722:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3782.761070] Lustre: server umount lustre-MDT0001 complete [ 3788.175403] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3789.183533] Key type lgssc unregistered [ 3789.288341] LNet: 2644:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3789.291623] LNet: Removed LNI 192.168.204.114@tcp [ 3812.660321] alg: No test for adler32 (adler32-zlib) [ 3813.415686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3813.534571] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 3813.613412] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3813.616367] LNet: Accept secure, port 988 [ 3815.171234] Key type lgssc registered [ 3815.512172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3819.296338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3819.313689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.512552] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.555865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.679474] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3824.977527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.170006] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.466197] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3830.369675] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3830.374970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.536794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.536335] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.544304] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:68 to 0x280000401:129 [ 3832.558005] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3834.541225] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.548064] LustreError: Skipped 1 previous similar message [ 3836.155191] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3836.162725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3836.304248] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3838.387161] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3839.589247] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:65 [ 3842.966318] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3848.020972] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3860.173194] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.178110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.187530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.343133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.344510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.351818] Lustre: Skipped 3 previous similar messages [ 3863.444015] Lustre: server umount lustre-MDT0000 complete [ 3864.830909] LustreError: 6583:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711095167 with bad export cookie 9580716049895703940 [ 3864.833648] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3864.843529] LustreError: 6583:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3864.993825] Lustre: server umount lustre-MDT0001 complete [ 3876.619811] Lustre: 10404:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711095169/real 1711095169] req@ffff8800a5000380 x1794213272379840/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711095179 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 3876.673204] Lustre: server umount lustre-OST0000 complete [ 3888.322565] Lustre: 10678:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711095180/real 1711095180] req@ffff8800a5000380 x1794213272380224/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711095190 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 3888.448123] Lustre: server umount lustre-OST0001 complete [ 3894.298192] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3898.439659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.643111] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.694322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.061285] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3903.692895] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.745075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.298764] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3909.328137] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3909.338398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3909.579500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.582656] Lustre: Skipped 1 previous similar message [ 3911.859216] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3913.582571] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.610279] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3915.616368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.728212] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3924.227112] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:97 [ 3924.232291] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:68 to 0x280000401:161 [ 3931.736079] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3933.463638] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3943.964608] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3943.969898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.979564] Lustre: Skipped 2 previous similar messages [ 3943.981923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3945.944840] Lustre: server umount lustre-MDT0000 complete [ 3947.410252] LustreError: 12054:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711095249 with bad export cookie 9580716049895705557 [ 3947.414645] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.420639] LustreError: 12054:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3947.555840] Lustre: server umount lustre-MDT0001 complete [ 3959.124643] Lustre: 16303:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711095251/real 1711095251] req@ffff88007b0a9500 x1794213272401280/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711095261 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 3959.167131] Lustre: server umount lustre-OST0000 complete [ 3970.865911] Lustre: 16577:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711095263/real 1711095263] req@ffff88007b0ab800 x1794213272401792/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711095273 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 3970.963390] Lustre: server umount lustre-OST0001 complete [ 3972.988944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 3973.571582] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 3975.671011] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 04:14:37 (1711095277) [ 3983.221618] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3986.310942] SGI XFS with ACLs, security attributes, no debug enabled [ 3986.324176] XFS (loop0): Mounting V5 Filesystem [ 3986.338790] XFS (loop0): Ending clean mount [ 3989.765051] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 3989.931013] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3996.963134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3999.878420] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4003.448383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4006.091853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.796845] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4008.806350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4011.726477] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4011.738974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4014.841400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4014.864045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.982606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4014.995174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4015.036316] Lustre: lustre-MDT0000: new disk, initializing [ 4015.075891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.078902] Lustre: Skipped 1 previous similar message [ 4015.092930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4016.462125] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4021.589046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4021.616609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.659004] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4021.689251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4021.692373] Lustre: Skipped 1 previous similar message [ 4021.732104] Lustre: lustre-MDT0001: new disk, initializing [ 4021.801887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4021.811360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4023.160962] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4026.350520] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4027.349979] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4030.333802] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4030.343064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.397934] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4030.407180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.584040] Lustre: lustre-OST0000: new disk, initializing [ 4030.588885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4032.114956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4032.122400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4032.169262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4032.813014] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4036.126907] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4037.569819] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4037.653849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4038.940088] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4039.038228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.157194] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.158731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.158732] Lustre: Skipped 3 previous similar messages [ 4042.159095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.159096] Lustre: Skipped 3 previous similar messages [ 4045.841619] Lustre: server umount lustre-OST0000 complete [ 4051.804791] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4051.811250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.819233] Lustre: Skipped 1 previous similar message [ 4051.823388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.826320] Lustre: Skipped 1 previous similar message [ 4053.654622] Lustre: server umount lustre-MDT0000 complete [ 4054.975954] LustreError: 25651:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711095357 with bad export cookie 9580716049895707083 [ 4054.978643] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.985355] LustreError: 25651:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4055.100448] Lustre: server umount lustre-MDT0001 complete [ 4058.002178] XFS (loop0): Unmounting Filesystem [ 4062.413196] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 04:16:04 (1711095364) [ 4065.902885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.128567] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.162660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.167623] Lustre: Skipped 2 previous similar messages [ 4067.465472] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4071.058495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.168465] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4071.171858] Lustre: Skipped 1 previous similar message [ 4072.541098] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4074.294595] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4075.254173] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4078.190266] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4078.199920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.397972] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4082.113363] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4096.396972] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.397209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.397211] Lustre: Skipped 1 previous similar message [ 4096.397809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4096.413957] LustreError: Skipped 1 previous similar message [ 4097.482428] Lustre: server umount lustre-OST0000 complete [ 4104.999947] Lustre: server umount lustre-MDT0000 complete [ 4106.412550] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.419808] LustreError: Skipped 1 previous similar message [ 4106.552750] LustreError: 27641:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711095409 with bad export cookie 9580716049895707930 [ 4106.553761] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4106.563790] LustreError: 27641:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4106.717956] Lustre: server umount lustre-MDT0001 complete [ 4111.155755] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4112.118826] Key type lgssc unregistered [ 4112.209214] LNet: 31414:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4112.212550] LNet: Removed LNI 192.168.204.114@tcp [ 4124.789086] alg: No test for adler32 (adler32-zlib) [ 4125.539663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4125.672453] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 4125.829638] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4125.831915] LNet: Accept secure, port 988 [ 4127.381687] Key type lgssc registered [ 4127.752250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4130.872340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4130.885715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.094010] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.114177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.478582] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4136.901985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.073941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4138.281935] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4139.940194] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4140.915378] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4143.774990] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4143.780353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.933779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.046442] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4148.120375] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4161.949485] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.955962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4161.958339] Lustre: Skipped 1 previous similar message [ 4163.444194] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 4166.956418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4166.961149] Lustre: Skipped 1 previous similar message [ 4167.039028] Lustre: server umount lustre-MDT0001 complete [ 4168.137756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4168.438853] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.281825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.347140] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4171.349916] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4171.367841] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4171.522854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4173.235218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4176.528378] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4181.071047] LustreError: 5165:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4181.076160] LustreError: 5165:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4181.080975] LustreError: 1078:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 4181.123081] LustreError: 5178:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 4181.132584] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4477.020920] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4477.024605] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.032818] Lustre: Skipped 1 previous similar message [ 4477.035257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4480.013115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.602836] Lustre: server umount lustre-MDT0001 complete [ 4482.661865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.942630] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.554534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.600858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4485.603678] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4485.629220] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4485.775111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4486.826999] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4490.783804] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4493.606119] LustreError: 21213:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4493.610311] LustreError: 21213:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4493.614357] LustreError: 1078:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 4493.617214] LustreError: 1078:0:(mgc_request.c:621:do_requeue()) Skipped 1 previous similar message [ 4493.638168] LustreError: 21224:0:(obd_config.c:703:class_setup()) Device 8 already setup (type osp) [ 4493.641479] LustreError: 21224:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 4493.649492] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.114@tcp [ 4791.262230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.268955] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4791.272423] Lustre: Skipped 4 previous similar messages [ 4795.690483] Lustre: server umount lustre-MDT0001 complete [ 4796.270521] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.276955] LustreError: Skipped 3 previous similar messages [ 4796.951430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.006264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.065593] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4800.068666] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4800.085984] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4800.211658] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4801.349761] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4805.217667] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4808.037806] LustreError: 4880:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4808.042169] LustreError: 4880:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4808.049111] LustreError: 1078:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 4808.053005] LustreError: 1078:0:(mgc_request.c:621:do_requeue()) Skipped 1 previous similar message [ 4808.084135] LustreError: 4885:0:(lod_lov.c:231:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4808.096227] LustreError: 4885:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -114 [ 4808.100300] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5115.678310] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5115.681729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.687181] Lustre: Skipped 1 previous similar message [ 5115.689882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5115.693974] Lustre: Skipped 1 previous similar message [ 5121.188682] Lustre: server umount lustre-MDT0000 complete [ 5122.788561] LustreError: 1065:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096425 with bad export cookie 7538664039018464207 [ 5122.789994] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.800661] LustreError: 1065:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5125.725163] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.725688] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.725690] Lustre: Skipped 2 previous similar messages [ 5125.742336] LustreError: Skipped 1 previous similar message [ 5128.949644] Lustre: server umount lustre-MDT0001 complete [ 5130.389218] Lustre: server umount lustre-OST0000 complete [ 5133.454901] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 5135.938518] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5139.140005] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5141.522854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5143.511042] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5143.518271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5145.809910] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5145.816258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5150.403812] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 04:34:12 (1711096452) [ 5152.801889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5152.833529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.935633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5152.946303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5152.985582] Lustre: lustre-MDT0000: new disk, initializing [ 5153.018724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5153.028333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5154.155030] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5158.573055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5158.600259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.635946] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5158.657161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5158.661358] Lustre: Skipped 1 previous similar message [ 5158.703472] Lustre: lustre-MDT0001: new disk, initializing [ 5158.741629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5158.761077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5158.766674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5159.954230] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5163.097021] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5164.076044] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5166.686531] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5166.691122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5166.715802] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5166.723582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5166.821290] Lustre: lustre-OST0000: new disk, initializing [ 5166.824800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5166.852163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.609273] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5168.880425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5168.888152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5168.922850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5171.600407] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5178.764935] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5178.768686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.775309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5178.778947] Lustre: Skipped 7 previous similar messages [ 5180.831295] Lustre: server umount lustre-MDT0000 complete [ 5182.454374] LustreError: 23857:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096484 with bad export cookie 7538664039018466104 [ 5182.454910] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5182.456094] LustreError: 24825:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 5182.456140] LustreError: 24825:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 5182.456145] LustreError: 24825:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 5182.477812] LustreError: 23857:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5182.629829] Lustre: server umount lustre-MDT0001 complete [ 5194.180679] Lustre: 27203:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711096486/real 1711096486] req@ffff8800768a4a80 x1794213599688320/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711096496 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5194.316254] Lustre: server umount lustre-OST0000 complete [ 5202.726425] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:35:04 (1711096504) [ 5209.774892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.956843] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.962956] LustreError: Skipped 1 previous similar message [ 5209.994405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.666418] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5215.249455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.633748] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5218.254435] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5219.384787] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5220.273490] Lustre: Failing over lustre-MDT0000 [ 5220.285493] LustreError: 29642:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5220.290460] LustreError: 29642:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5220.294393] LustreError: 29642:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 5220.363520] Lustre: server umount lustre-MDT0000 complete [ 5220.380542] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5220.384862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.392384] Lustre: Skipped 2 previous similar messages [ 5230.428407] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.433966] LustreError: Skipped 6 previous similar messages [ 5233.387493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.454810] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.544671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.547864] Lustre: Skipped 1 previous similar message [ 5233.564875] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5234.525755] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5237.408313] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 5238.540188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5238.541659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5238.555913] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5239.555075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.558041] Lustre: Skipped 3 previous similar messages [ 5245.586881] Lustre: server umount lustre-MDT0000 complete [ 5247.145501] LustreError: 29596:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096549 with bad export cookie 7538664039018467973 [ 5247.147157] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5247.156945] LustreError: 29596:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5247.313872] Lustre: server umount lustre-MDT0001 complete [ 5250.712191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.924137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.016880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5255.278835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.636971] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5258.225063] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5259.169376] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.048866] Lustre: Failing over lustre-MDT0000 [ 5260.051429] LustreError: 729:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5260.059151] LustreError: 729:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5260.063404] LustreError: 729:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5260.139833] Lustre: server umount lustre-MDT0000 complete [ 5260.428396] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5265.481048] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.488898] LustreError: Skipped 4 previous similar messages [ 5273.589061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.650892] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5273.796357] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5274.923383] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5277.736881] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 5278.781211] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5278.783078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5278.783079] Lustre: Skipped 1 previous similar message [ 5278.808838] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5285.833706] Lustre: server umount lustre-MDT0000 complete [ 5287.466580] LustreError: 2231:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096589 with bad export cookie 7538664039018468883 [ 5287.469681] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.475061] LustreError: 2231:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5287.634378] Lustre: server umount lustre-MDT0001 complete [ 5293.771326] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:36:35 (1711096595) [ 5300.740692] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 5304.118562] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5307.985238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5310.867842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5313.597317] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5313.605615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5316.351362] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5316.359793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5319.220834] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5323.101190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5323.129794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.230864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5323.243293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5323.281467] Lustre: lustre-MDT0000: new disk, initializing [ 5323.314565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5323.317481] Lustre: Skipped 2 previous similar messages [ 5323.325100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5324.702052] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5329.976069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.003929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5330.057721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5330.060589] Lustre: Skipped 1 previous similar message [ 5330.098920] Lustre: lustre-MDT0001: new disk, initializing [ 5330.156985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5330.166066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5331.359915] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5334.530300] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5335.452377] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5338.194444] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5338.205846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5338.232591] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5338.239306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.364747] Lustre: lustre-OST0000: new disk, initializing [ 5338.368762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5340.113763] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5342.316379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5342.320811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5342.340819] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 5342.363215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5342.371121] LustreError: 10257:0:(ofd_dev.c:1593:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x280000401:33, last_id 65281. Likely MDS last_id corruption [ 5342.386532] LustreError: 10875:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -22 [ 5342.391436] LustreError: 10875:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -22 [ 5343.162987] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.631293] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 5352.381158] LustreError: 10257:0:(ofd_dev.c:1593:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x280000401:33, last_id 65281. Likely MDS last_id corruption [ 5352.386886] LustreError: 10875:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -22 [ 5352.392050] LustreError: 10875:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -22 [ 5357.373473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.374118] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.374854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.374855] Lustre: Skipped 6 previous similar messages [ 5357.403460] Lustre: Skipped 8 previous similar messages [ 5359.249211] Lustre: server umount lustre-OST0000 complete [ 5366.997224] Lustre: server umount lustre-MDT0000 complete [ 5367.389220] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5367.394247] LustreError: Skipped 3 previous similar messages [ 5368.217881] LustreError: 9199:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096670 with bad export cookie 7538664039018469268 [ 5368.219068] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.227866] LustreError: 9199:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5368.384309] Lustre: server umount lustre-MDT0001 complete [ 5371.103901] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5371.760563] Key type lgssc unregistered [ 5371.837019] LNet: 12382:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5371.840453] LNet: Removed LNI 192.168.204.114@tcp [ 5383.194455] alg: No test for adler32 (adler32-zlib) [ 5383.945622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5384.038193] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 5384.119407] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5384.122811] LNet: Accept secure, port 988 [ 5385.675680] Key type lgssc registered [ 5386.038252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5389.446731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5389.455067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.662631] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.701247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.040685] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5396.300386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.499370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5398.074801] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5400.329800] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5401.389251] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5404.315264] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5404.322206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.487819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.488146] Lustre: lustre-OST0000: deleting orphan objects from 0x280000402:747 to 0x280000402:833 [ 5406.524189] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5408.024988] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5426.173720] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:38:47 (1711096727) [ 5427.090709] Lustre: Modifying parameter general.jobid_name in log params [ 5428.035694] Lustre: Modifying parameter general.jobid_name in log params [ 5433.544888] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:38:55 (1711096735) [ 5439.849919] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:39:01 (1711096741) [ 5440.736431] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [ 5447.293167] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:39:08 (1711096748) [ 5448.186753] Lustre: Setting parameter general.osc.*.max_pages_per_rpc in log params [ 5448.189222] Lustre: Skipped 2 previous similar messages [ 5456.688359] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc in log lustre-client [ 5456.691543] Lustre: Skipped 7 previous similar messages [ 5460.788254] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:39:22 (1711096762) [ 5462.157835] Lustre: *** cfs_fail_loc=131b, val=2*** [ 5463.924884] Lustre: *** cfs_fail_loc=131b, val=2*** [ 5471.023911] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:39:32 (1711096772) [ 5471.818472] Lustre: Modifying parameter general.jobid_name in log params [ 5471.822940] Lustre: Skipped 3 previous similar messages [ 5479.894348] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:39:41 (1711096781) [ 5486.605521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.606590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5486.616142] Lustre: Skipped 1 previous similar message [ 5490.790688] Lustre: server umount lustre-OST0000 complete [ 5496.620545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.628955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5496.631750] Lustre: Skipped 2 previous similar messages [ 5498.692316] Lustre: server umount lustre-MDT0000 complete [ 5500.569921] LustreError: 14265:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096803 with bad export cookie 8037654250196415988 [ 5500.574020] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.584159] LustreError: 14265:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5500.832900] Lustre: server umount lustre-MDT0001 complete [ 5505.848057] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5506.889379] Key type lgssc unregistered [ 5507.088461] LNet: 26313:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5507.094287] LNet: Removed LNI 192.168.204.114@tcp [ 5521.871421] alg: No test for adler32 (adler32-zlib) [ 5522.628816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5522.724636] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 5522.794422] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5522.796591] LNet: Accept secure, port 988 [ 5524.338862] Key type lgssc registered [ 5524.692239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5528.067914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5531.389906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5534.581952] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5534.589156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5538.069691] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5538.082679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5541.975233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5541.998404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5542.006466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.140851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5543.160259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5543.205277] Lustre: lustre-MDT0000: new disk, initializing [ 5543.249039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.260289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5544.678923] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5550.056370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5550.090120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.128054] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5550.159710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5550.162805] Lustre: Skipped 1 previous similar message [ 5550.208532] Lustre: lustre-MDT0001: new disk, initializing [ 5550.245352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.270240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5550.276578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5550.304936] mount.lustre (30977) used greatest stack depth: 9488 bytes left [ 5551.645803] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5554.931889] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5555.988458] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5558.711369] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5558.721236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5558.746542] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5558.752991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5558.856192] Lustre: lustre-OST0000: new disk, initializing [ 5558.860643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5558.892603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.095363] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5564.116667] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5565.284235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5565.290258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5565.319262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5565.492017] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5565.561911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5566.756802] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5566.831227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5570.316940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.317356] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5570.318056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5570.333756] Lustre: Skipped 1 previous similar message [ 5573.705901] Lustre: server umount lustre-OST0000 complete [ 5575.324978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.330372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5575.333102] Lustre: Skipped 1 previous similar message [ 5580.288277] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5580.291848] LustreError: Skipped 1 previous similar message [ 5580.294266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.300448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.372822] Lustre: server umount lustre-MDT0000 complete [ 5582.786842] LustreError: 31001:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096885 with bad export cookie 3741520044701886172 [ 5582.787877] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.796520] LustreError: 31001:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5582.911882] Lustre: server umount lustre-MDT0001 complete [ 5590.639466] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:41:32 (1711096892) [ 5594.232312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.497543] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.542847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.281379] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5599.550271] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5600.973126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.171257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5602.814194] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5604.855220] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5605.890103] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5609.700740] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5609.723592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.010808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5612.476736] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5614.731962] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5622.029066] LustreError: 5315:0:(llog.c:436:llog_init_handle()) llog type is not specified! [ 5622.609651] Lustre: Modifying parameter general.timeout in log params [ 5624.886283] Lustre: Modifying parameter general.timeout in log params [ 5624.893277] Lustre: Skipped 3 previous similar messages [ 5629.048596] Lustre: Modifying parameter general.timeout in log params [ 5629.051083] Lustre: Skipped 7 previous similar messages [ 5637.135817] Lustre: Modifying parameter general.timeout in log params [ 5637.148792] Lustre: Skipped 14 previous similar messages [ 5653.427909] Lustre: Modifying parameter general.timeout in log params [ 5653.432235] Lustre: Skipped 36 previous similar messages [ 5684.023604] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:43:05 (1711096985) [ 5691.282088] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5691.286621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.294320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.302640] Lustre: Skipped 1 previous similar message [ 5694.462906] Lustre: server umount lustre-MDT0000 complete [ 5696.620324] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 5696.625675] LustreError: 2790:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711096999 with bad export cookie 3741520044701887033 [ 5696.625914] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.865619] Lustre: server umount lustre-MDT0001 complete [ 5709.165066] Lustre: 15976:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711097001/real 1711097001] req@ffff880136a62680 x1794215064417664/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711097011 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5719.187684] Lustre: 15976:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711097011/real 1711097011] req@ffff880136a62680 x1794215064417856/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711097021 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5719.337192] Lustre: server umount lustre-OST0000 complete [ 5723.810358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5725.895048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5727.795715] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5727.807317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5729.817044] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5729.831005] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5733.364804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.518908] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5733.546854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5733.549791] Lustre: Skipped 66 previous similar messages [ 5733.701719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.428653] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5740.009849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.108414] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5741.958327] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5744.279113] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5745.549107] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5749.207124] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5749.216384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5749.351020] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5752.050807] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5754.472269] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5760.173550] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5760.177112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.191663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5764.680353] Lustre: server umount lustre-OST0000 complete [ 5770.221405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.226823] Lustre: Skipped 1 previous similar message [ 5770.230138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.235774] Lustre: Skipped 2 previous similar messages [ 5772.880552] Lustre: server umount lustre-MDT0000 complete [ 5774.914956] LustreError: 17887:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097077 with bad export cookie 3741520044701889182 [ 5774.920288] LustreError: 17887:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5774.920984] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5775.114637] Lustre: server umount lustre-MDT0001 complete [ 5780.838338] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5782.140157] Key type lgssc unregistered [ 5782.261567] LNet: 21838:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5782.271573] LNet: Removed LNI 192.168.204.114@tcp [ 5798.699368] alg: No test for adler32 (adler32-zlib) [ 5799.452831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5799.595978] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 5799.735097] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5799.741248] LNet: Accept secure, port 988 [ 5801.298434] Key type lgssc registered [ 5801.758371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5805.772193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5805.784802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.000091] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.059084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.924120] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5813.349555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.675165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.341810] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5817.517636] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5818.800375] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5822.652855] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5822.662327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5822.990144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.637881] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5827.778947] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5836.352397] Lustre: Modifying parameter general.jobid_var in log params [ 5842.015538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.016753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5842.026235] Lustre: Skipped 2 previous similar messages [ 5845.923341] Lustre: server umount lustre-MDT0000 complete [ 5847.022429] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.035352] LustreError: Skipped 2 previous similar messages [ 5847.853391] LustreError: 23726:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097150 with bad export cookie 5500737787684714081 [ 5847.854135] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.873532] LustreError: 23726:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5848.016025] Lustre: server umount lustre-MDT0001 complete [ 5860.052889] Lustre: 27452:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711097152/real 1711097152] req@ffff8801301f8000 x1794215354842496/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711097162 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5860.208572] Lustre: server umount lustre-OST0000 complete [ 5864.101946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.693400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5867.439237] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5867.446251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5869.089197] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5869.098729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5871.919112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.103072] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5872.127454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5872.251300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.008245] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5878.308317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.381597] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5878.405945] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5878.669964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5880.364474] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5882.448396] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5883.552423] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5886.726534] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5886.733612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5886.833142] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5886.972345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.178193] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5891.397427] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5892.338332] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5897.917924] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5897.924638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.941600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5897.948533] Lustre: Skipped 2 previous similar messages [ 5901.733860] Lustre: server umount lustre-OST0000 complete [ 5907.964352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5907.965966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5907.973690] Lustre: Skipped 2 previous similar messages [ 5909.792121] Lustre: server umount lustre-MDT0000 complete [ 5911.886150] LustreError: 29354:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097214 with bad export cookie 5500737787684714914 [ 5911.889644] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5911.899756] LustreError: 29354:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5912.090167] Lustre: server umount lustre-MDT0001 complete [ 5917.098280] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5918.372400] Key type lgssc unregistered [ 5918.524243] LNet: 864:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5918.532312] LNet: Removed LNI 192.168.204.114@tcp [ 5969.908789] alg: No test for adler32 (adler32-zlib) [ 5970.667082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5970.833861] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 5970.960037] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5970.963267] LNet: Accept secure, port 988 [ 5972.573663] Key type lgssc registered [ 5973.232065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5973.830491] LustreError: 6993:0:(lu_object.c:1416:lu_context_key_register()) cfs_race id 60d sleeping [ 5982.685468] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5987.124550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5995.310902] LustreError: 6993:0:(lu_object.c:1416:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 5995.331744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.632185] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.692433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.424998] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6006.030410] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:48:27 (1711097307) [ 6006.695818] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.247879] LustreError: 8151:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6008.252451] LustreError: 8151:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6008.257104] LustreError: 8151:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 6008.437795] Lustre: server umount lustre-MDT0000 complete [ 6013.998994] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6014.786832] Key type lgssc unregistered [ 6014.882422] LNet: 10235:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6014.887025] LNet: Removed LNI 192.168.204.114@tcp [ 6021.073406] alg: No test for adler32 (adler32-zlib) [ 6021.824665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6021.911253] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 6021.969174] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6021.971141] LNet: Accept secure, port 988 [ 6023.514465] Key type lgssc registered [ 6023.842206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.638998] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.650484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.842699] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.882348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.252264] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6032.536131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.789002] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6034.017564] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6035.690940] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6036.561453] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6039.373668] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6039.383186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.612252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.592490] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6043.326235] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6081.916695] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:49:43 (1711097383) [ 6082.543708] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 6085.742251] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:49:47 (1711097387) [ 6087.661272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.662630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.676108] Lustre: Skipped 2 previous similar messages [ 6092.671028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.673407] Lustre: Skipped 2 previous similar messages [ 6093.126695] Lustre: server umount lustre-MDT0000 complete [ 6094.566889] LustreError: 15373:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097397 with bad export cookie 16320857038428469004 [ 6094.568147] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.580253] LustreError: 15373:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6094.728463] Lustre: server umount lustre-MDT0001 complete [ 6106.212857] Lustre: 15910:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711097398/real 1711097398] req@ffff88009e039500 x1794215587629184/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711097408 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6106.419861] Lustre: server umount lustre-OST0000 complete [ 6111.425157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.632147] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.721742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.005242] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6116.280348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.532993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.627729] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6119.216775] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6120.077390] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6122.331337] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6122.342934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6125.121339] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6125.130579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6125.161087] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6125.168288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.264920] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 6125.268944] LustreError: 16731:0:(mgs_handler.c:515:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 6125.290333] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 6125.295348] LustreError: 18928:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -98 [ 6125.300658] LustreError: 18928:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 6125.303634] LustreError: 18928:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 6125.374049] Lustre: server umount lustre-OST0000 complete [ 6125.378402] LustreError: 18928:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 6128.565573] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6128.574420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6128.652704] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 6128.657063] LustreError: 16730:0:(mgs_handler.c:515:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 6128.666313] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 6128.670639] LustreError: 19542:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -98 [ 6128.675014] LustreError: 19542:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 6128.678109] LustreError: 19542:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 6128.732437] Lustre: server umount lustre-OST0000 complete [ 6128.734431] LustreError: 19542:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 6129.782419] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6129.790764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.302381] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6132.309663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6132.406378] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6132.409169] Lustre: Found index 0 for lustre-OST0000, updating log [ 6132.413405] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 6132.437141] Lustre: lustre-OST0000: new disk, initializing [ 6132.440565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6132.516185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.311742] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6135.887133] Lustre: Failing over lustre-OST0000 [ 6135.969517] Lustre: server umount lustre-OST0000 complete [ 6140.514012] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.514582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6140.514583] Lustre: Skipped 2 previous similar messages [ 6140.534524] Lustre: Skipped 1 previous similar message [ 6143.523707] Lustre: server umount lustre-MDT0000 complete [ 6145.509719] LustreError: 19561:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097448 with bad export cookie 16320857038428471755 [ 6145.515214] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6145.771978] Lustre: server umount lustre-MDT0001 complete [ 6152.079478] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:50:53 (1711097453) [ 6158.010853] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6161.961140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.177689] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.238156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.584764] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6167.246790] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.471376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.229432] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6173.388126] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6173.400989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.761620] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6179.317707] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6179.324894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6179.366131] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6179.375942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.411860] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6179.444572] Lustre: lustre-OST0001: new disk, initializing [ 6179.447464] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6181.658366] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6183.238741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6183.242925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6183.274823] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6190.454975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6194.752798] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 6201.821535] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6201.825853] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.833499] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6201.836591] Lustre: Skipped 1 previous similar message [ 6206.620138] Lustre: server umount lustre-MDT0001 complete [ 6207.295120] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.301332] LustreError: Skipped 2 previous similar messages [ 6207.862989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6210.869905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.942383] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6210.944999] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6210.969922] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6211.209560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6211.215271] Lustre: Skipped 3 previous similar messages [ 6212.548163] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6212.889178] LustreError: 28245:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 6212.892627] LustreError: 28245:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 6212.898524] LustreError: 23454:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 6212.936471] LustreError: 28258:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 6212.941866] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6216.210957] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 6219.284021] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 04:52:00 (1711097520) [ 6225.202258] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6241.391186] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6242.807034] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 6274.435386] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6274.438692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.444250] Lustre: Skipped 2 previous similar messages [ 6274.448979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6274.451305] Lustre: Skipped 3 previous similar messages [ 6275.492283] Lustre: server umount lustre-MDT0000 complete [ 6276.300804] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6276.308979] LustreError: Skipped 4 previous similar messages [ 6277.116694] LustreError: 27025:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097579 with bad export cookie 16320857038428472651 [ 6277.117612] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6277.126367] LustreError: 27025:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6277.297547] Lustre: server umount lustre-MDT0001 complete [ 6288.918631] Lustre: 958:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711097581/real 1711097581] req@ffff88013586c700 x1794215587736576/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711097591 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6288.977044] Lustre: server umount lustre-OST0000 complete [ 6300.419624] Lustre: 1232:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711097592/real 1711097592] req@ffff88009e03bb80 x1794215587737088/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711097602 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6300.516824] Lustre: server umount lustre-OST0001 complete [ 6302.643272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6306.039240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6306.412167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6309.470301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6312.663871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.008579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6319.184313] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6322.199440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.207525] Lustre: lustre-MDT0000: reset Object Index mappings [ 6322.378825] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.410704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.435639] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6326.247298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.259365] Lustre: lustre-MDT0001: reset Object Index mappings [ 6327.446171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6330.190010] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6330.195005] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6331.868373] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6332.375662] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:323 to 0x280000401:354 [ 6334.656783] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6334.662921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6335.822186] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:194 to 0x280000400:225 [ 6335.823559] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:194 to 0x2c0000400:225 [ 6335.854653] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:162 to 0x2c0000401:193 [ 6336.477308] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6339.731109] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6342.364879] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 6368.099706] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:54:30 (1711097670) [ 6380.892895] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.894011] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6380.899599] Lustre: Skipped 5 previous similar messages [ 6385.444822] Lustre: server umount lustre-MDT0000 complete [ 6385.900306] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.905737] LustreError: Skipped 4 previous similar messages [ 6386.625921] LustreError: 4876:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711097689 with bad export cookie 16320857038428518949 [ 6386.627161] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6386.635620] LustreError: 4876:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6390.908574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.914079] Lustre: Skipped 2 previous similar messages [ 6392.777551] Lustre: server umount lustre-MDT0001 complete [ 6400.123335] Lustre: server umount lustre-OST0000 complete [ 6407.583932] Lustre: server umount lustre-OST0001 complete [ 6409.374911] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6411.720615] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6415.175167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6417.777564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6420.346414] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6420.352338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6422.677995] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6422.684715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.206560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.235854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.333311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6425.350736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6425.392446] Lustre: lustre-MDT0000: new disk, initializing [ 6425.428618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6426.551349] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6430.185972] Lustre: server umount lustre-MDT0000 complete [ 6431.469954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.284670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.322555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.471341] Lustre: Found index 0 for lustre-MDT0000, updating log [ 6434.480772] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6435.958035] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6438.165097] Lustre: server umount lustre-MDT0000 complete [ 6444.239250] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:55:45 (1711097745) [ 6444.835591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 6447.660132] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:55:49 (1711097749) [ 6474.833816] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:56:16 (1711097776) [ 6480.748251] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6483.426903] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6486.547010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6489.242376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6491.419561] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6491.425811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6493.580365] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6493.587648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6496.009125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6496.031048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.127370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6496.129537] Lustre: Skipped 1 previous similar message [ 6496.138931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6496.141851] Lustre: Skipped 1 previous similar message [ 6496.178878] Lustre: lustre-MDT0000: new disk, initializing [ 6496.212319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.215445] Lustre: Skipped 5 previous similar messages [ 6496.224349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6497.344738] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6501.699725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6501.720821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.762743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6501.765354] Lustre: Skipped 1 previous similar message [ 6501.802390] Lustre: lustre-MDT0001: new disk, initializing [ 6501.844089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6501.848577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6502.855899] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6505.729696] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6506.654070] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6509.179093] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6509.189615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6509.222145] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6509.229816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.510709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6510.515548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6510.545280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6511.237888] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6514.164909] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6517.868235] Lustre: lustre-MDD0000: changelog on [ 6518.989156] Lustre: lustre-MDD0001: changelog on [ 6694.396956] Lustre: 28213:0:(llog_cat.c:939:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6716.933357] Lustre: lustre-MDD0001: changelog off [ 6718.329943] Lustre: lustre-MDD0000: changelog off [ 6720.861031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.863940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6720.863942] Lustre: Skipped 5 previous similar messages [ 6720.874257] Lustre: Skipped 1 previous similar message [ 6726.349521] Lustre: server umount lustre-OST0000 complete [ 6733.946316] Lustre: server umount lustre-MDT0000 complete [ 6735.669963] LustreError: 24610:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711098038 with bad export cookie 16320857038428557561 [ 6735.671337] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6735.680947] LustreError: 24610:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6735.839890] Lustre: server umount lustre-MDT0001 complete [ 6740.673027] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6742.045671] Key type lgssc unregistered [ 6742.157377] LNet: 31292:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6742.160544] LNet: Removed LNI 192.168.204.114@tcp [ 6760.791198] alg: No test for adler32 (adler32-zlib) [ 6761.547641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6761.648407] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 6761.736683] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6761.739102] LNet: Accept secure, port 988 [ 6763.289110] Key type lgssc registered [ 6763.724133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6768.262575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6771.562229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6774.863110] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6774.871188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6778.017997] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6778.026180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.472368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.513342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6781.526215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.670423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6782.683272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6782.721295] Lustre: lustre-MDT0000: new disk, initializing [ 6782.762119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6782.780762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6784.325905] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6789.485088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.523175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.564152] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6789.588406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6789.591418] Lustre: Skipped 1 previous similar message [ 6789.631601] Lustre: lustre-MDT0001: new disk, initializing [ 6789.680684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.706411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6789.714328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6790.985567] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6794.028349] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6795.026992] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6797.949664] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6797.957046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6797.987852] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6797.998327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.117876] Lustre: lustre-OST0000: new disk, initializing [ 6798.121370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6798.160461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.291634] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6803.452051] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6804.776774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6804.779778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6804.830810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6810.629375] LustreError: 5864:0:(echo_client.c:689:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6810.633760] LustreError: 5864:0:(obd_config.c:797:class_setup()) setup ec failed (-22) [ 6811.663883] LustreError: 5993:0:(echo_client.c:1896:echo_md_handler()) Can not resolve the path : rc = -2 [ 6819.708757] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6819.713733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.723659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6822.873677] Lustre: server umount lustre-MDT0000 complete [ 6824.527567] LustreError: 4009:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711098127 with bad export cookie 9518082916457270835 [ 6824.530190] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.537130] LustreError: 4009:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6824.704722] Lustre: server umount lustre-MDT0001 complete [ 6836.666691] Lustre: 6988:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711098129/real 1711098129] req@ffff880077cf4380 x1794216363572864/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711098139 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6836.771343] Lustre: server umount lustre-OST0000 complete [ 6846.952615] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6850.510954] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6854.890816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6857.733729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6860.679051] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6860.687592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6863.433942] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6863.441933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6866.175651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6866.207339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.324324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6866.341994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6866.379151] Lustre: lustre-MDT0000: new disk, initializing [ 6866.416324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.423628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6867.532158] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6873.122559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.162677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.223393] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6873.258029] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6873.262518] Lustre: Skipped 1 previous similar message [ 6873.313460] Lustre: lustre-MDT0001: new disk, initializing [ 6873.381468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6873.402694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6873.408595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6874.795076] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6878.062731] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6879.065863] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6882.049790] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6882.056525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.086790] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6882.093850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.193453] Lustre: lustre-OST0000: new disk, initializing [ 6882.196200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6882.231093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.118807] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6887.011492] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6888.353263] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6889.440260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6889.447656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6889.485813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6890.474585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6891.670198] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6891.755336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6894.477824] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6894.479657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.479659] Lustre: Skipped 2 previous similar messages [ 6894.480123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6894.480124] Lustre: Skipped 3 previous similar messages [ 6894.501503] LustreError: Skipped 1 previous similar message [ 6898.676205] Lustre: server umount lustre-OST0000 complete [ 6903.404942] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.411148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.416444] Lustre: Skipped 1 previous similar message [ 6903.426030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.428494] Lustre: Skipped 1 previous similar message [ 6906.729306] Lustre: server umount lustre-MDT0000 complete [ 6908.734209] LustreError: 10652:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711098211 with bad export cookie 9518082916457271752 [ 6908.735556] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.748557] LustreError: 10652:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6908.969523] Lustre: server umount lustre-MDT0001 complete [ 6913.678998] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:03:35 (1711098215) [ 6917.315459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.606455] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.682136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6919.091056] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6922.627821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.653325] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.123228] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6925.838454] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6926.752862] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6929.378816] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6929.383881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6931.471400] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6933.013120] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6947.565956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.567516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6947.567522] Lustre: Skipped 2 previous similar messages [ 6947.587129] Lustre: Skipped 3 previous similar messages [ 6951.975360] Lustre: server umount lustre-MDT0000 complete [ 6952.572639] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.581755] LustreError: Skipped 2 previous similar messages [ 6953.664969] LustreError: 15646:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711098256 with bad export cookie 9518082916457272592 [ 6953.665591] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6953.675494] LustreError: 15646:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6957.585872] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.590585] LustreError: Skipped 1 previous similar message [ 6957.590991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.592414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6957.592415] Lustre: Skipped 3 previous similar messages [ 6959.939340] Lustre: server umount lustre-MDT0001 complete [ 6963.644322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6969.713584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.255389] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6978.265777] LustreError: Skipped 2 previous similar messages [ 6988.267070] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.277599] LustreError: Skipped 3 previous similar messages [ 7004.295768] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7004.305436] LustreError: Skipped 15 previous similar messages [ 7029.939943] LustreError: 13a-8: Failed to get MGS log lustre-mdtir and no local copy. [ 7029.950374] Lustre: 19890:0:(mgc_request.c:1844:mgc_process_log()) MGC192.168.204.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7038.993177] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.003523] LustreError: Skipped 23 previous similar messages [ 7058.291882] LustreError: 13a-8: Failed to get MGS log lustre-mdtir and no local copy. [ 7058.296230] Lustre: 15658:0:(mgc_request.c:1844:mgc_process_log()) MGC192.168.204.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7069.964499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7069.970075] Lustre: Skipped 2 previous similar messages [ 7071.335028] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7073.967606] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7075.326371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.300734] LustreError: 15658:0:(mgc_request.c:621:do_requeue()) failed processing log: -5 [ 7099.004367] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7100.617399] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7102.842266] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7103.635898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7104.165968] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 7105.022632] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7105.813068] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7116.536873] LustreError: 21661:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff880130818400) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7116.558767] Lustre: server umount lustre-MDT0000 complete [ 7118.844902] LustreError: 15647:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711098421 with bad export cookie 9518082916457279095 [ 7118.848040] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 7118.850304] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7118.869121] LustreError: 15647:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7119.177201] Lustre: server umount lustre-MDT0001 complete [ 7131.207670] Lustre: 22194:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711098423/real 1711098423] req@ffff8800a1ced180 x1794216363618112/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711098433 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7141.223653] Lustre: 22194:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711098433/real 1711098433] req@ffff8800a1ced180 x1794216363618368/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711098443 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7141.388881] Lustre: server umount lustre-OST0000 complete [ 7152.140071] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7155.258699] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7159.782859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7164.297172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7167.928110] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7167.936232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7171.328387] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7171.339572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7175.072418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7175.103030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.220382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7175.236879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7175.285011] Lustre: lustre-MDT0000: new disk, initializing [ 7175.322989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.327735] Lustre: Skipped 1 previous similar message [ 7175.338272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7176.584054] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7181.662017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7181.701806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.894091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7183.600331] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7187.121650] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7188.227797] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7191.785494] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7191.794739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7191.825321] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7191.834630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.460323] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7198.175606] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.736248] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7201.464594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7201.516391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7201.919164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7203.387583] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7203.491449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7206.510975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.511075] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7206.511076] LustreError: Skipped 1 previous similar message [ 7206.512302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7206.512303] Lustre: Skipped 2 previous similar messages [ 7206.535423] Lustre: Skipped 1 previous similar message [ 7210.486174] Lustre: server umount lustre-OST0000 complete [ 7218.275536] Lustre: server umount lustre-MDT0000 complete [ 7219.583879] LustreError: 26832:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711098522 with bad export cookie 9518082916457295888 [ 7219.586361] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.592973] LustreError: 26832:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7219.711650] Lustre: server umount lustre-MDT0001 complete [ 7225.415446] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:08:47 (1711098527) [ 7228.442804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.663990] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.673127] LustreError: Skipped 45 previous similar messages [ 7229.807335] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7233.138145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.418572] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7236.036172] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7237.190553] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7240.356933] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7240.366608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.627978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7244.631576] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7244.889774] Lustre: MGS: Client 0f2cf2cf-30c1-47ff-9e32-74cc7a0292d0 (at 192.168.204.14@tcp) reconnecting [ 7245.967544] Lustre: DEBUG MARKER: conf-sanity test_150: @@@@@@ FAIL: client start failed