== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:27:07 (1744021627) start mds service on oleg232-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg232-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1744021680.26 total 10.00 last 23.40) - open/close 498 (time 1744021690.30 total 20.05 last 26.27) - open/close 806 (time 1744021701.11 total 30.86 last 28.50) - open/close 1069 (time 1744021711.79 total 41.53 last 24.64) - open/close 1311 (time 1744021722.44 total 52.18 last 22.72) - open/close 1697 (time 1744021733.30 total 63.04 last 35.54) - open/close 2066 (time 1744021744.19 total 73.93 last 33.88) - open/close 2266 (time 1744021756.70 total 86.44 last 15.99) - open/close 2352 (time 1744021766.93 total 96.67 last 8.41) - open/close 2783 (time 1744021777.91 total 107.66 last 39.23) - open/close 3069 (time 1744021788.85 total 118.59 last 26.15) - open/close 3287 (time 1744021799.65 total 129.39 last 20.19) - open/close 3398 (time 1744021810.16 total 139.90 last 10.56) - open/close 3468 (time 1744021820.50 total 150.25 last 6.77) - open/close 3602 (time 1744021831.13 total 160.87 last 12.61) - open/close 3789 (time 1744021841.84 total 171.58 last 17.45) - open/close 3885 (time 1744021852.27 total 182.01 last 9.21) - open/close 3945 (time 1744021862.57 total 192.31 last 5.83) - open/close 4150 (time 1744021873.55 total 203.29 last 18.68) - open/close 4330 (time 1744021885.40 total 215.14 last 15.19) - open/close 4376 (time 1744021895.62 total 225.36 last 4.50) - open/close 4566 (time 1744021906.44 total 236.19 last 17.56) - open/close 4688 (time 1744021917.00 total 246.74 last 11.56) - open/close 4828 (time 1744021927.63 total 257.37 last 13.17) - open/close 5050 (time 1744021938.46 total 268.20 last 20.49) - open/close 5200 (time 1744021949.13 total 278.88 last 14.05) - open/close 5291 (time 1744021959.56 total 289.31 last 8.72) - open/close 5432 (time 1744021970.25 total 299.99 last 13.20) - open/close 5726 (time 1744021980.25 total 309.99 last 29.40) - open/close 8773 (time 1744021990.25 total 320.00 last 304.65) - open/close 10000 (time 1744021994.98 total 324.73 last 259.39) - open/close 12370 (time 1744022004.98 total 334.73 last 236.98) - open/close 14806 (time 1744022014.98 total 344.73 last 243.57) - open/close 18190 (time 1744022024.99 total 354.73 last 338.34) - open/close 20000 (time 1744022030.36 total 360.10 last 337.14) - open/close 22488 (time 1744022040.36 total 370.10 last 248.75) - open/close 25430 (time 1744022050.36 total 380.10 last 294.11) - open/close 27842 (time 1744022060.36 total 390.11 last 241.14) - open/close 30000 (time 1744022066.96 total 396.71 last 326.91) - open/close 33115 (time 1744022076.97 total 406.71 last 311.46) - open/close 36364 (time 1744022086.97 total 416.71 last 324.81) - open/close 39050 (time 1744022096.97 total 426.71 last 268.59) - open/close 40000 (time 1744022099.28 total 429.02 last 411.30) - open/close 44440 (time 1744022109.28 total 439.02 last 443.93) - open/close 46889 (time 1744022119.28 total 449.02 last 244.90) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 457.86 seconds: 108.40 ops/second - unlinked 0 (time 1744022129 ; total 0 ; last 0) - unlinked 10000 (time 1744022148 ; total 19 ; last 19) - unlinked 20000 (time 1744022166 ; total 37 ; last 18) - unlinked 30000 (time 1744022181 ; total 52 ; last 15) - unlinked 40000 (time 1744022204 ; total 75 ; last 23) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 98 seconds: 506.448975 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2