== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:34:49 (1746142489) start mds service on oleg255-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 256 (time 1746142545.98 total 10.47 last 24.45) - open/close 576 (time 1746142556.78 total 21.27 last 29.63) - open/close 706 (time 1746142566.79 total 31.27 last 12.99) - open/close 873 (time 1746142577.07 total 41.55 last 16.25) - open/close 1057 (time 1746142587.44 total 51.93 last 17.73) - open/close 1303 (time 1746142598.36 total 62.84 last 22.54) - open/close 1570 (time 1746142608.90 total 73.39 last 25.32) - open/close 1718 (time 1746142619.84 total 84.33 last 13.52) - open/close 1865 (time 1746142630.52 total 95.00 last 13.78) - open/close 2112 (time 1746142640.98 total 105.47 last 23.60) - open/close 2314 (time 1746142651.44 total 115.92 last 19.32) - open/close 2583 (time 1746142663.90 total 128.39 last 21.58) - open/close 2777 (time 1746142674.50 total 138.98 last 18.31) - open/close 3017 (time 1746142684.50 total 148.99 last 23.99) - open/close 3231 (time 1746142696.50 total 160.98 last 17.84) - open/close 3465 (time 1746142706.52 total 171.00 last 23.36) - open/close 3780 (time 1746142717.40 total 181.88 last 28.95) - open/close 4011 (time 1746142727.40 total 191.88 last 23.10) - open/close 4232 (time 1746142737.40 total 201.89 last 22.09) - open/close 4536 (time 1746142748.13 total 212.61 last 28.34) - open/close 4801 (time 1746142760.47 total 224.95 last 21.48) - open/close 5026 (time 1746142770.47 total 234.96 last 22.49) - open/close 5243 (time 1746142780.64 total 245.12 last 21.35) - open/close 5540 (time 1746142790.64 total 255.13 last 29.68) - open/close 5700 (time 1746142801.35 total 265.83 last 14.95) - open/close 5940 (time 1746142811.78 total 276.26 last 23.01) - open/close 6151 (time 1746142821.78 total 286.26 last 21.10) - open/close 6339 (time 1746142832.34 total 296.82 last 17.80) - open/close 6642 (time 1746142842.34 total 306.82 last 30.30) - open/close 6887 (time 1746142852.43 total 316.92 last 24.28) - open/close 7163 (time 1746142862.87 total 327.36 last 26.43) - open/close 7335 (time 1746142873.63 total 338.12 last 15.98) - open/close 7559 (time 1746142884.17 total 348.65 last 21.27) - open/close 7872 (time 1746142894.86 total 359.35 last 29.27) - open/close 8131 (time 1746142905.03 total 369.51 last 25.48) - open/close 8411 (time 1746142915.45 total 379.94 last 26.86) - open/close 8621 (time 1746142926.37 total 390.86 last 19.23) - open/close 8790 (time 1746142936.38 total 400.87 last 16.88) - open/close 9071 (time 1746142947.38 total 411.87 last 25.54) - open/close 9338 (time 1746142958.08 total 422.56 last 24.97) - open/close 9521 (time 1746142968.08 total 432.57 last 18.28) - open/close 9796 (time 1746142978.80 total 443.29 last 25.65) - open/close 10000 (time 1746142984.20 total 448.68 last 37.82) - open/close 11363 (time 1746142994.21 total 458.70 last 136.14) - open/close 12712 (time 1746143004.22 total 468.70 last 134.80) - open/close 14267 (time 1746143014.22 total 478.70 last 155.46) - open/close 15599 (time 1746143024.22 total 488.71 last 133.14) - open/close 16832 (time 1746143034.23 total 498.71 last 123.24) - open/close 18211 (time 1746143044.23 total 508.72 last 137.87) - open/close 19452 (time 1746143054.25 total 518.73 last 123.92) - open/close 20000 (time 1746143058.33 total 522.82 last 134.07) - open/close 21303 (time 1746143068.33 total 532.82 last 130.29) - open/close 22575 (time 1746143078.34 total 542.82 last 127.15) - open/close 23767 (time 1746143088.34 total 552.83 last 119.17) - open/close 24985 (time 1746143098.35 total 562.83 last 121.71) - open/close 26263 (time 1746143108.35 total 572.83 last 127.79) - open/close 27491 (time 1746143118.35 total 582.83 last 122.80) - open/close 28766 (time 1746143128.35 total 592.84 last 127.43) - open/close 30000 (time 1746143137.30 total 601.78 last 137.95) - open/close 31257 (time 1746143147.31 total 611.79 last 125.62) - open/close 32373 (time 1746143157.31 total 621.80 last 111.50) - open/close 33419 (time 1746143167.33 total 631.81 last 104.47) - open/close 34542 (time 1746143177.34 total 641.83 last 112.15) - open/close 35654 (time 1746143187.34 total 651.83 last 111.15) - open/close 36935 (time 1746143197.35 total 661.84 last 128.00) - open/close 38096 (time 1746143207.36 total 671.85 last 115.98) - open/close 39295 (time 1746143217.37 total 681.85 last 119.83) - open/close 40000 (time 1746143223.45 total 687.93 last 116.01) - open/close 41225 (time 1746143233.45 total 697.94 last 122.43) - open/close 42517 (time 1746143243.46 total 707.95 last 129.07) - open/close 43811 (time 1746143253.47 total 717.95 last 129.34) - open/close 45118 (time 1746143263.47 total 727.95 last 130.68) - open/close 46370 (time 1746143273.47 total 737.95 last 125.17) - open/close 47583 (time 1746143283.47 total 747.96 last 121.26) - open/close 48848 (time 1746143293.48 total 757.97 last 126.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 763.80 seconds: 64.98 ops/second - unlinked 0 (time 1746143301 ; total 0 ; last 0) - unlinked 10000 (time 1746143364 ; total 63 ; last 63) - unlinked 20000 (time 1746143412 ; total 111 ; last 48) - unlinked 30000 (time 1746143460 ; total 159 ; last 48) - unlinked 40000 (time 1746143505 ; total 204 ; last 45) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 248 seconds: 200.129028 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2