== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:14:20 (1767320060) start mds service on oleg307-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 165 (time 1767320251.89 total 10.06 last 16.41) - open/close 300 (time 1767320262.05 total 20.22 last 13.28) - open/close 484 (time 1767320272.06 total 30.23 last 18.38) - open/close 620 (time 1767320282.08 total 40.25 last 13.58) - open/close 722 (time 1767320292.55 total 50.72 last 9.74) - open/close 980 (time 1767320303.00 total 61.17 last 24.69) - open/close 1107 (time 1767320313.17 total 71.34 last 12.49) - open/close 1286 (time 1767320323.92 total 82.09 last 16.65) - open/close 1520 (time 1767320333.92 total 92.09 last 23.39) - open/close 1635 (time 1767320343.93 total 102.10 last 11.49) - open/close 1793 (time 1767320354.60 total 112.77 last 14.81) - open/close 1859 (time 1767320364.60 total 122.77 last 6.60) - open/close 1940 (time 1767320378.66 total 136.82 last 5.76) - open/close 2154 (time 1767320388.66 total 146.83 last 21.39) - open/close 2270 (time 1767320399.44 total 157.61 last 10.76) - open/close 2377 (time 1767320410.38 total 168.55 last 9.79) - open/close 2423 (time 1767320422.03 total 180.20 last 3.95) - open/close 2516 (time 1767320432.47 total 190.64 last 8.91) - open/close 2628 (time 1767320443.17 total 201.34 last 10.47) - open/close 2876 (time 1767320454.18 total 212.35 last 22.52) - open/close 2994 (time 1767320464.78 total 222.95 last 11.13) - open/close 3279 (time 1767320474.81 total 232.98 last 28.43) - open/close 3531 (time 1767320485.73 total 243.90 last 23.07) - open/close 3723 (time 1767320495.74 total 253.91 last 19.19) - open/close 3911 (time 1767320505.74 total 263.91 last 18.80) - open/close 4028 (time 1767320516.77 total 274.94 last 10.61) - open/close 4135 (time 1767320527.20 total 285.37 last 10.26) - open/close 4340 (time 1767320537.45 total 295.62 last 20.00) - open/close 4420 (time 1767320547.80 total 305.97 last 7.73) - open/close 4678 (time 1767320557.83 total 316.00 last 25.73) - open/close 4805 (time 1767320567.84 total 326.01 last 12.69) - open/close 4909 (time 1767320578.59 total 336.75 last 9.68) - open/close 5123 (time 1767320590.95 total 349.12 last 17.31) - open/close 5375 (time 1767320600.95 total 359.12 last 25.19) - open/close 5608 (time 1767320610.98 total 369.15 last 23.25) - open/close 5793 (time 1767320620.98 total 379.15 last 18.50) - open/close 5964 (time 1767320631.05 total 389.22 last 16.97) - open/close 6131 (time 1767320643.68 total 401.85 last 13.23) - open/close 6390 (time 1767320654.60 total 412.77 last 23.72) - open/close 6634 (time 1767320666.44 total 424.61 last 20.60) - open/close 6821 (time 1767320676.47 total 434.64 last 18.65) - open/close 7052 (time 1767320687.15 total 445.32 last 21.63) - open/close 7307 (time 1767320697.15 total 455.32 last 25.49) - open/close 7507 (time 1767320707.26 total 465.43 last 19.79) - open/close 7688 (time 1767320717.27 total 475.44 last 18.08) - open/close 7867 (time 1767320727.85 total 486.02 last 16.93) - open/close 8034 (time 1767320738.63 total 496.80 last 15.49) - open/close 8276 (time 1767320748.63 total 506.80 last 24.19) - open/close 8479 (time 1767320758.65 total 516.82 last 20.27) - open/close 8739 (time 1767320769.58 total 527.75 last 23.79) - open/close 9005 (time 1767320780.20 total 538.37 last 25.05) - open/close 9182 (time 1767320790.20 total 548.37 last 17.69) - open/close 9326 (time 1767320800.21 total 558.38 last 14.38) - open/close 9488 (time 1767320810.23 total 568.40 last 16.16) - open/close 9682 (time 1767320820.25 total 578.42 last 19.37) - open/close 9929 (time 1767320830.80 total 588.97 last 23.42) - open/close 10000 (time 1767320833.65 total 591.82 last 24.85) - open/close 10086 (time 1767320843.86 total 602.03 last 8.43) - open/close 10248 (time 1767320854.10 total 612.27 last 15.81) - open/close 10561 (time 1767320864.88 total 623.05 last 29.03) - open/close 10770 (time 1767320875.28 total 633.45 last 20.10) - open/close 10941 (time 1767320885.31 total 643.48 last 17.06) - open/close 11085 (time 1767320895.47 total 653.63 last 14.18) - open/close 11325 (time 1767320905.48 total 663.65 last 23.97) - open/close 11568 (time 1767320915.81 total 673.98 last 23.53) - open/close 11839 (time 1767320925.81 total 683.98 last 27.08) - open/close 12087 (time 1767320935.82 total 693.99 last 24.79) - open/close 12328 (time 1767320948.17 total 706.34 last 19.50) - open/close 12598 (time 1767320958.68 total 716.85 last 25.69) - open/close 12790 (time 1767320968.70 total 726.87 last 19.16) - open/close 13053 (time 1767320978.72 total 736.89 last 26.26) - open/close 13254 (time 1767320988.76 total 746.93 last 20.02) - open/close 13461 (time 1767320999.07 total 757.24 last 20.08) - open/close 13651 (time 1767321014.05 total 772.22 last 12.68) - open/close 13881 (time 1767321026.23 total 784.40 last 18.88) - open/close 14127 (time 1767321037.30 total 795.47 last 22.22) - open/close 14285 (time 1767321047.60 total 805.77 last 15.34) - open/close 14482 (time 1767321058.57 total 816.74 last 17.96) - open/close 14633 (time 1767321068.63 total 826.79 last 15.02) - open/close 14943 (time 1767321079.38 total 837.55 last 28.83) - open/close 15204 (time 1767321089.39 total 847.56 last 26.08) - open/close 15476 (time 1767321099.40 total 857.57 last 27.17) - open/close 15652 (time 1767321109.42 total 867.59 last 17.56) - open/close 15896 (time 1767321120.09 total 878.26 last 22.86) - open/close 16145 (time 1767321130.10 total 888.27 last 24.88) - open/close 16350 (time 1767321140.56 total 898.73 last 19.61) - open/close 16645 (time 1767321151.49 total 909.66 last 26.98) - open/close 16899 (time 1767321162.24 total 920.41 last 23.63) - open/close 17070 (time 1767321173.16 total 931.33 last 15.66) - open/close 17315 (time 1767321183.56 total 941.73 last 23.55) - open/close 17558 (time 1767321194.20 total 952.37 last 22.85) - open/close 17739 (time 1767321204.65 total 962.82 last 17.32) - open/close 17937 (time 1767321214.65 total 972.82 last 19.80) - open/close 18217 (time 1767321227.17 total 985.34 last 22.36) - open/close 18465 (time 1767321237.97 total 996.14 last 22.96) - open/close 18699 (time 1767321248.58 total 1006.75 last 22.05) - open/close 19414 (time 1767321258.59 total 1016.76 last 71.49) - open/close 20000 (time 1767321266.58 total 1024.75 last 73.29) - open/close 20787 (time 1767321276.59 total 1034.76 last 78.62) - open/close 21652 (time 1767321286.59 total 1044.76 last 86.49) - open/close 22494 (time 1767321296.61 total 1054.78 last 84.07) - open/close 23382 (time 1767321306.62 total 1064.79 last 88.73) - open/close 24260 (time 1767321316.62 total 1074.79 last 87.77) - open/close 25156 (time 1767321326.62 total 1084.79 last 89.59) - open/close 25972 (time 1767321336.62 total 1094.79 last 81.59) - open/close 26878 (time 1767321346.62 total 1104.79 last 90.57) - open/close 27728 (time 1767321356.62 total 1114.79 last 85.00) - open/close 28509 (time 1767321366.63 total 1124.80 last 78.09) - open/close 29357 (time 1767321376.63 total 1134.80 last 84.79) - open/close 30000 (time 1767321384.04 total 1142.21 last 86.74) - open/close 30911 (time 1767321394.05 total 1152.21 last 91.05) - open/close 31808 (time 1767321404.05 total 1162.22 last 89.65) - open/close 32567 (time 1767321414.05 total 1172.22 last 75.88) - open/close 33275 (time 1767321424.06 total 1182.23 last 70.78) - open/close 33891 (time 1767321434.07 total 1192.24 last 61.50) - open/close 34553 (time 1767321444.08 total 1202.25 last 66.17) - open/close 35319 (time 1767321454.08 total 1212.25 last 76.56) - open/close 36134 (time 1767321464.09 total 1222.26 last 81.45) - open/close 36901 (time 1767321474.10 total 1232.27 last 76.61) - open/close 37712 (time 1767321484.11 total 1242.28 last 81.00) - open/close 38411 (time 1767321494.13 total 1252.30 last 69.74) - open/close 39191 (time 1767321504.14 total 1262.31 last 77.96) - open/close 40000 (time 1767321514.07 total 1272.24 last 81.48) - open/close 40786 (time 1767321524.07 total 1282.24 last 78.55) - open/close 41622 (time 1767321534.08 total 1292.25 last 83.55) - open/close 42375 (time 1767321544.09 total 1302.26 last 75.23) - open/close 43216 (time 1767321554.09 total 1312.26 last 84.09) - open/close 44043 (time 1767321564.09 total 1322.26 last 82.68) - open/close 44817 (time 1767321574.10 total 1332.27 last 77.34) - open/close 45549 (time 1767321584.11 total 1342.28 last 73.14) - open/close 46364 (time 1767321594.12 total 1352.29 last 81.42) - open/close 47113 (time 1767321604.12 total 1362.29 last 74.87) - open/close 47917 (time 1767321614.14 total 1372.31 last 80.24) - open/close 48676 (time 1767321624.15 total 1382.32 last 75.85) - open/close 49375 (time 1767321634.16 total 1392.33 last 69.86) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1395.36 seconds: 35.57 ops/second - unlinked 0 (time 1767321641 ; total 0 ; last 0) - unlinked 10000 (time 1767321768 ; total 127 ; last 127) - unlinked 20000 (time 1767321891 ; total 250 ; last 123) - unlinked 30000 (time 1767322012 ; total 371 ; last 121) - unlinked 40000 (time 1767322139 ; total 498 ; last 127) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 620 seconds: 80.051613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2