== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:21:12 (1763349672) start mds service on oleg425-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg425-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 79 (time 1763349767.09 total 10.71 last 7.38) - open/close 295 (time 1763349777.10 total 20.71 last 21.58) - open/close 423 (time 1763349788.07 total 31.69 last 11.67) - open/close 654 (time 1763349798.09 total 41.71 last 23.05) - open/close 871 (time 1763349808.09 total 51.71 last 21.69) - open/close 1085 (time 1763349819.10 total 62.72 last 19.45) - open/close 1309 (time 1763349829.67 total 73.29 last 21.19) - open/close 1539 (time 1763349839.68 total 83.29 last 22.98) - open/close 1741 (time 1763349850.14 total 93.75 last 19.31) - open/close 1917 (time 1763349860.16 total 103.77 last 17.57) - open/close 2119 (time 1763349870.16 total 113.78 last 20.19) - open/close 2326 (time 1763349881.06 total 124.67 last 19.00) - open/close 2543 (time 1763349891.88 total 135.50 last 20.04) - open/close 2817 (time 1763349901.89 total 145.51 last 27.38) - open/close 3022 (time 1763349911.89 total 155.51 last 20.50) - open/close 3249 (time 1763349922.08 total 165.70 last 22.29) - open/close 3474 (time 1763349932.09 total 175.71 last 22.47) - open/close 3692 (time 1763349942.24 total 185.86 last 21.48) - open/close 3935 (time 1763349952.80 total 196.42 last 23.01) - open/close 4125 (time 1763349962.81 total 206.42 last 18.99) - open/close 4326 (time 1763349973.72 total 217.34 last 18.41) - open/close 4562 (time 1763349984.02 total 227.64 last 22.91) - open/close 4751 (time 1763349995.04 total 238.66 last 17.15) - open/close 4995 (time 1763350005.40 total 249.02 last 23.57) - open/close 5226 (time 1763350016.24 total 259.85 last 21.31) - open/close 5462 (time 1763350026.24 total 269.86 last 23.59) - open/close 5634 (time 1763350036.26 total 279.87 last 17.17) - open/close 5805 (time 1763350046.26 total 289.88 last 17.09) - open/close 5986 (time 1763350056.27 total 299.89 last 18.09) - open/close 6166 (time 1763350066.33 total 309.95 last 17.89) - open/close 6407 (time 1763350076.52 total 320.13 last 23.66) - open/close 6585 (time 1763350086.63 total 330.25 last 17.60) - open/close 6794 (time 1763350096.63 total 340.25 last 20.89) - open/close 7007 (time 1763350107.10 total 350.72 last 20.34) - open/close 7236 (time 1763350117.12 total 360.73 last 22.86) - open/close 7425 (time 1763350127.71 total 371.33 last 17.83) - open/close 7652 (time 1763350137.74 total 381.35 last 22.65) - open/close 7843 (time 1763350147.86 total 391.48 last 18.86) - open/close 8037 (time 1763350157.88 total 401.50 last 19.37) - open/close 8214 (time 1763350167.89 total 411.50 last 17.69) - open/close 8431 (time 1763350177.90 total 421.51 last 21.67) - open/close 8651 (time 1763350187.94 total 431.55 last 21.92) - open/close 8891 (time 1763350198.63 total 442.25 last 22.43) - open/close 9066 (time 1763350209.07 total 452.69 last 16.77) - open/close 9277 (time 1763350219.07 total 462.69 last 21.10) - open/close 9474 (time 1763350229.63 total 473.25 last 18.65) - open/close 9733 (time 1763350239.64 total 483.26 last 25.89) - open/close 10000 (time 1763350249.33 total 492.95 last 27.54) - open/close 10222 (time 1763350260.01 total 503.63 last 20.80) - open/close 10398 (time 1763350271.01 total 514.63 last 16.00) - open/close 10560 (time 1763350281.01 total 524.63 last 16.19) - open/close 10785 (time 1763350291.25 total 534.86 last 21.99) - open/close 10926 (time 1763350301.26 total 544.87 last 14.09) - open/close 11129 (time 1763350311.28 total 554.89 last 20.26) - open/close 11347 (time 1763350321.89 total 565.50 last 20.55) - open/close 11557 (time 1763350332.45 total 576.07 last 19.87) - open/close 11816 (time 1763350342.83 total 586.45 last 24.96) - open/close 12061 (time 1763350352.83 total 596.45 last 24.49) - open/close 12272 (time 1763350363.30 total 606.92 last 20.16) - open/close 12523 (time 1763350373.32 total 616.94 last 25.05) - open/close 12726 (time 1763350383.51 total 627.13 last 19.92) - open/close 12982 (time 1763350393.84 total 637.46 last 24.79) - open/close 13231 (time 1763350404.51 total 648.13 last 23.34) - open/close 13493 (time 1763350414.52 total 658.13 last 26.18) - open/close 13722 (time 1763350424.92 total 668.54 last 22.01) - open/close 13996 (time 1763350435.89 total 679.51 last 24.97) - open/close 14213 (time 1763350446.68 total 690.29 last 20.13) - open/close 14467 (time 1763350456.69 total 700.31 last 25.36) - open/close 14694 (time 1763350466.91 total 710.52 last 22.22) - open/close 14919 (time 1763350477.26 total 720.88 last 21.73) - open/close 15129 (time 1763350487.29 total 730.90 last 20.94) - open/close 15363 (time 1763350497.50 total 741.12 last 22.92) - open/close 15583 (time 1763350507.50 total 751.12 last 21.99) - open/close 15777 (time 1763350517.55 total 761.17 last 19.31) - open/close 16029 (time 1763350528.51 total 772.13 last 22.98) - open/close 16335 (time 1763350538.53 total 782.15 last 30.56) - open/close 16539 (time 1763350548.67 total 792.28 last 20.12) - open/close 16777 (time 1763350559.46 total 803.08 last 22.05) - open/close 16996 (time 1763350569.47 total 813.09 last 21.89) - open/close 17211 (time 1763350579.55 total 823.16 last 21.34) - open/close 17845 (time 1763350589.55 total 833.17 last 63.37) - open/close 18587 (time 1763350599.57 total 843.19 last 74.07) - open/close 19339 (time 1763350609.57 total 853.19 last 75.18) - open/close 20000 (time 1763350619.03 total 862.65 last 69.86) - open/close 20808 (time 1763350629.04 total 872.65 last 80.76) - open/close 21613 (time 1763350639.04 total 882.66 last 80.45) - open/close 22397 (time 1763350649.05 total 892.66 last 78.38) - open/close 23207 (time 1763350659.06 total 902.68 last 80.90) - open/close 23973 (time 1763350669.07 total 912.68 last 76.55) - open/close 24741 (time 1763350679.07 total 922.69 last 76.75) - open/close 25426 (time 1763350689.08 total 932.70 last 68.45) - open/close 26207 (time 1763350699.08 total 942.70 last 78.06) - open/close 26931 (time 1763350709.09 total 952.71 last 72.34) - open/close 27687 (time 1763350719.10 total 962.72 last 75.55) - open/close 28448 (time 1763350729.10 total 972.72 last 76.10) - open/close 29226 (time 1763350739.11 total 982.73 last 77.71) - open/close 29913 (time 1763350749.11 total 992.73 last 68.70) - open/close 30638 (time 1763350759.12 total 1002.74 last 72.42) - open/close 31450 (time 1763350769.13 total 1012.75 last 81.14) - open/close 32238 (time 1763350779.14 total 1022.76 last 78.72) - open/close 32981 (time 1763350789.15 total 1032.76 last 74.25) - open/close 33637 (time 1763350799.15 total 1042.77 last 65.58) - open/close 34227 (time 1763350809.15 total 1052.77 last 58.97) - open/close 34846 (time 1763350819.16 total 1062.77 last 61.89) - open/close 35561 (time 1763350829.17 total 1072.79 last 71.40) - open/close 36236 (time 1763350839.18 total 1082.80 last 67.41) - open/close 36924 (time 1763350849.18 total 1092.80 last 68.79) - open/close 37573 (time 1763350859.19 total 1102.81 last 64.83) - open/close 38289 (time 1763350869.20 total 1112.81 last 71.60) - open/close 39055 (time 1763350879.20 total 1122.81 last 76.59) - open/close 39816 (time 1763350889.20 total 1132.82 last 76.08) - open/close 40000 (time 1763350891.95 total 1135.57 last 66.94) - open/close 40634 (time 1763350901.96 total 1145.58 last 63.33) - open/close 41289 (time 1763350911.96 total 1155.58 last 65.50) - open/close 42008 (time 1763350921.96 total 1165.58 last 71.88) - open/close 42824 (time 1763350931.96 total 1175.58 last 81.59) - open/close 43605 (time 1763350941.98 total 1185.60 last 77.98) - open/close 44332 (time 1763350952.00 total 1195.61 last 72.58) - open/close 45062 (time 1763350962.02 total 1205.64 last 72.82) - open/close 45772 (time 1763350972.03 total 1215.65 last 70.94) - open/close 46387 (time 1763350982.04 total 1225.65 last 61.45) - open/close 47093 (time 1763350992.05 total 1235.67 last 70.50) - open/close 47791 (time 1763351002.06 total 1245.68 last 69.74) - open/close 48479 (time 1763351012.07 total 1255.68 last 68.76) - open/close 49169 (time 1763351022.08 total 1265.70 last 68.91) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1272.20 seconds: 39.01 ops/second - unlinked 0 (time 1763351033 ; total 0 ; last 0) - unlinked 10000 (time 1763351166 ; total 133 ; last 133) - unlinked 20000 (time 1763351297 ; total 264 ; last 131) - unlinked 30000 (time 1763351436 ; total 403 ; last 139) - unlinked 40000 (time 1763351560 ; total 527 ; last 124) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 655 seconds: 75.774048 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg425-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2