== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:31:16 (1763404276) start mds service on oleg157-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1763404352.39 total 10.02 last 18.26) - open/close 412 (time 1763404362.40 total 20.03 last 22.87) - open/close 582 (time 1763404372.42 total 30.05 last 16.97) - open/close 757 (time 1763404383.02 total 40.65 last 16.51) - open/close 993 (time 1763404393.02 total 50.65 last 23.60) - open/close 1236 (time 1763404403.75 total 61.38 last 22.64) - open/close 1430 (time 1763404413.76 total 71.40 last 19.38) - open/close 1639 (time 1763404423.78 total 81.41 last 20.87) - open/close 1801 (time 1763404434.21 total 91.85 last 15.52) - open/close 1989 (time 1763404445.20 total 102.83 last 17.12) - open/close 2184 (time 1763404455.51 total 113.14 last 18.91) - open/close 2377 (time 1763404466.45 total 124.08 last 17.64) - open/close 2630 (time 1763404476.45 total 134.09 last 25.28) - open/close 2864 (time 1763404486.46 total 144.09 last 23.40) - open/close 3070 (time 1763404496.63 total 154.26 last 20.24) - open/close 3289 (time 1763404506.63 total 164.27 last 21.89) - open/close 3478 (time 1763404516.64 total 174.27 last 18.89) - open/close 3691 (time 1763404527.63 total 185.26 last 19.37) - open/close 3916 (time 1763404537.83 total 195.46 last 22.06) - open/close 4165 (time 1763404548.75 total 206.38 last 22.81) - open/close 4377 (time 1763404558.76 total 216.39 last 21.17) - open/close 4539 (time 1763404569.11 total 226.74 last 15.65) - open/close 4774 (time 1763404579.32 total 236.95 last 23.02) - open/close 4964 (time 1763404590.36 total 247.99 last 17.21) - open/close 5193 (time 1763404600.36 total 257.99 last 22.90) - open/close 5401 (time 1763404611.19 total 268.82 last 19.20) - open/close 5583 (time 1763404621.78 total 279.41 last 17.18) - open/close 5784 (time 1763404632.09 total 289.72 last 19.50) - open/close 6024 (time 1763404642.10 total 299.73 last 23.97) - open/close 6160 (time 1763404652.33 total 309.96 last 13.29) - open/close 6373 (time 1763404662.57 total 320.20 last 20.81) - open/close 6603 (time 1763404673.25 total 330.89 last 21.52) - open/close 6835 (time 1763404683.92 total 341.55 last 21.76) - open/close 7054 (time 1763404699.55 total 357.18 last 14.01) - open/close 7148 (time 1763404709.99 total 367.62 last 9.01) - open/close 7313 (time 1763404719.99 total 377.62 last 16.50) - open/close 7535 (time 1763404730.84 total 388.47 last 20.46) - open/close 7738 (time 1763404741.06 total 398.70 last 19.85) - open/close 7905 (time 1763404751.73 total 409.36 last 15.66) - open/close 8175 (time 1763404761.75 total 419.38 last 26.94) - open/close 8335 (time 1763404772.45 total 430.08 last 14.95) - open/close 8451 (time 1763404782.47 total 440.10 last 11.58) - open/close 8568 (time 1763404792.49 total 450.12 last 11.68) - open/close 8655 (time 1763404802.50 total 460.13 last 8.69) - open/close 8813 (time 1763404814.68 total 472.31 last 12.97) - open/close 8992 (time 1763404824.75 total 482.38 last 17.78) - open/close 9108 (time 1763404835.12 total 492.75 last 11.19) - open/close 9177 (time 1763404845.77 total 503.40 last 6.48) - open/close 9306 (time 1763404856.14 total 513.77 last 12.44) - open/close 9478 (time 1763404866.85 total 524.48 last 16.06) - open/close 9664 (time 1763404877.13 total 534.76 last 18.08) - open/close 9831 (time 1763404888.85 total 546.49 last 14.25) - open/close 9973 (time 1763404899.87 total 557.51 last 12.89) - open/close 10189 (time 1763404910.56 total 568.19 last 20.22) - open/close 10410 (time 1763404921.29 total 578.92 last 20.60) - open/close 10652 (time 1763404931.31 total 588.94 last 24.15) - open/close 10890 (time 1763404942.06 total 599.69 last 22.13) - open/close 11131 (time 1763404952.63 total 610.26 last 22.81) - open/close 11339 (time 1763404963.12 total 620.75 last 19.81) - open/close 11522 (time 1763404973.13 total 630.76 last 18.29) - open/close 11728 (time 1763404983.25 total 640.88 last 20.36) - open/close 11981 (time 1763404993.38 total 651.01 last 24.98) - open/close 12218 (time 1763405003.93 total 661.56 last 22.45) - open/close 12429 (time 1763405014.98 total 672.61 last 19.10) - open/close 12687 (time 1763405025.00 total 682.63 last 25.76) - open/close 12853 (time 1763405035.80 total 693.43 last 15.37) - open/close 13063 (time 1763405045.81 total 703.44 last 20.99) - open/close 13274 (time 1763405056.83 total 714.46 last 19.14) - open/close 13522 (time 1763405066.83 total 724.47 last 24.78) - open/close 13684 (time 1763405077.61 total 735.25 last 15.03) - open/close 13909 (time 1763405087.76 total 745.40 last 22.17) - open/close 14133 (time 1763405098.52 total 756.15 last 20.83) - open/close 14338 (time 1763405110.80 total 768.43 last 16.69) - open/close 14554 (time 1763405120.92 total 778.56 last 21.33) - open/close 14771 (time 1763405130.94 total 788.57 last 21.66) - open/close 14899 (time 1763405140.96 total 798.60 last 12.77) - open/close 15084 (time 1763405150.97 total 808.60 last 18.49) - open/close 15298 (time 1763405161.45 total 819.08 last 20.42) - open/close 15502 (time 1763405171.46 total 829.09 last 20.38) - open/close 15769 (time 1763405181.60 total 839.23 last 26.34) - open/close 15988 (time 1763405192.02 total 849.65 last 21.02) - open/close 16226 (time 1763405202.96 total 860.59 last 21.75) - open/close 16441 (time 1763405213.02 total 870.66 last 21.36) - open/close 16612 (time 1763405223.04 total 880.67 last 17.07) - open/close 16722 (time 1763405233.05 total 890.69 last 10.99) - open/close 16906 (time 1763405243.06 total 900.70 last 18.38) - open/close 17110 (time 1763405254.03 total 911.66 last 18.60) - open/close 17338 (time 1763405264.54 total 922.17 last 21.69) - open/close 17784 (time 1763405274.55 total 932.19 last 44.55) - open/close 18479 (time 1763405284.56 total 942.19 last 69.47) - open/close 19162 (time 1763405294.56 total 952.19 last 68.29) - open/close 19854 (time 1763405304.58 total 962.21 last 69.06) - open/close 20000 (time 1763405306.67 total 964.30 last 69.82) - open/close 20671 (time 1763405316.68 total 974.32 last 67.00) - open/close 21313 (time 1763405326.69 total 984.33 last 64.14) - open/close 22000 (time 1763405336.72 total 994.36 last 68.50) - open/close 22638 (time 1763405346.73 total 1004.36 last 63.75) - open/close 23332 (time 1763405356.74 total 1014.38 last 69.32) - open/close 24021 (time 1763405366.75 total 1024.38 last 68.88) - open/close 24669 (time 1763405376.75 total 1034.39 last 64.75) - open/close 25339 (time 1763405386.76 total 1044.40 last 66.94) - open/close 25989 (time 1763405396.77 total 1054.40 last 64.99) - open/close 26696 (time 1763405406.77 total 1064.40 last 70.65) - open/close 27323 (time 1763405416.79 total 1074.42 last 62.61) - open/close 27936 (time 1763405426.79 total 1084.43 last 61.26) - open/close 28593 (time 1763405436.80 total 1094.43 last 65.66) - open/close 29248 (time 1763405446.80 total 1104.44 last 65.47) - open/close 29881 (time 1763405456.81 total 1114.44 last 63.25) - open/close 30506 (time 1763405466.82 total 1124.45 last 62.47) - open/close 31162 (time 1763405476.83 total 1134.46 last 65.53) - open/close 31808 (time 1763405486.84 total 1144.47 last 64.55) - open/close 32438 (time 1763405496.85 total 1154.48 last 62.92) - open/close 33106 (time 1763405506.85 total 1164.48 last 66.79) - open/close 33763 (time 1763405516.85 total 1174.49 last 65.67) - open/close 34360 (time 1763405526.86 total 1184.49 last 59.68) - open/close 34990 (time 1763405536.86 total 1194.49 last 62.99) - open/close 35638 (time 1763405546.87 total 1204.50 last 64.74) - open/close 36278 (time 1763405556.88 total 1214.51 last 63.94) - open/close 36942 (time 1763405566.89 total 1224.52 last 66.32) - open/close 37600 (time 1763405576.91 total 1234.54 last 65.70) - open/close 38207 (time 1763405586.92 total 1244.55 last 60.63) - open/close 38858 (time 1763405596.93 total 1254.56 last 65.04) - open/close 39519 (time 1763405606.93 total 1264.56 last 66.09) - open/close 40000 (time 1763405613.84 total 1271.47 last 69.63) - open/close 40673 (time 1763405623.84 total 1281.47 last 67.27) - open/close 41354 (time 1763405633.85 total 1291.48 last 68.05) - open/close 42008 (time 1763405643.86 total 1301.49 last 65.32) - open/close 42688 (time 1763405653.88 total 1311.52 last 67.83) - open/close 43401 (time 1763405663.89 total 1321.52 last 71.29) - open/close 44078 (time 1763405673.89 total 1331.52 last 67.68) - open/close 44772 (time 1763405683.91 total 1341.54 last 69.28) - open/close 45464 (time 1763405693.91 total 1351.54 last 69.18) - open/close 46152 (time 1763405703.92 total 1361.55 last 68.74) - open/close 46793 (time 1763405713.93 total 1371.56 last 64.05) - open/close 47444 (time 1763405723.93 total 1381.57 last 65.06) - open/close 48099 (time 1763405733.94 total 1391.57 last 65.48) - open/close 48722 (time 1763405743.94 total 1401.57 last 62.29) - open/close 49405 (time 1763405753.94 total 1411.57 last 68.27) total: 50000 open/close in 1420.47 seconds: 35.20 ops/second - unlinked 0 (time 1763405766 ; total 0 ; last 0) - unlinked 10000 (time 1763405923 ; total 157 ; last 157) - unlinked 20000 (time 1763406091 ; total 325 ; last 168) - unlinked 30000 (time 1763406249 ; total 483 ; last 158) - unlinked 40000 (time 1763406406 ; total 640 ; last 157) total: 50000 unlinks in 803 seconds: 62.266502 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg157-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2