== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:23:00 (1769077380) start mds service on oleg241-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 100 (time 1769077473.53 total 10.26 last 9.75) - open/close 199 (time 1769077485.41 total 22.14 last 8.33) - open/close 400 (time 1769077495.43 total 32.17 last 20.04) - open/close 583 (time 1769077505.44 total 42.17 last 18.29) - open/close 782 (time 1769077515.88 total 52.62 last 19.05) - open/close 946 (time 1769077527.26 total 63.99 last 14.42) - open/close 1181 (time 1769077537.27 total 74.00 last 23.49) - open/close 1289 (time 1769077548.33 total 85.06 last 9.76) - open/close 1393 (time 1769077558.35 total 95.08 last 10.38) - open/close 1500 (time 1769077569.19 total 105.92 last 9.87) - open/close 1676 (time 1769077579.76 total 116.49 last 16.65) - open/close 1799 (time 1769077590.29 total 127.03 last 11.67) - open/close 2032 (time 1769077600.41 total 137.14 last 23.04) - open/close 2316 (time 1769077610.41 total 147.14 last 28.39) - open/close 2510 (time 1769077620.44 total 157.17 last 19.35) - open/close 2726 (time 1769077633.43 total 170.16 last 16.63) - open/close 2934 (time 1769077644.32 total 181.05 last 19.10) - open/close 3105 (time 1769077654.65 total 191.38 last 16.56) - open/close 3322 (time 1769077664.98 total 201.71 last 20.99) - open/close 3499 (time 1769077674.98 total 211.72 last 17.70) - open/close 3710 (time 1769077685.46 total 222.20 last 20.14) - open/close 3951 (time 1769077695.49 total 232.23 last 24.03) - open/close 4152 (time 1769077705.50 total 242.23 last 20.09) - open/close 4258 (time 1769077716.25 total 252.98 last 9.86) - open/close 4473 (time 1769077726.26 total 262.99 last 21.49) - open/close 4600 (time 1769077738.64 total 275.37 last 10.25) - open/close 4718 (time 1769077749.21 total 285.94 last 11.16) - open/close 4934 (time 1769077759.90 total 296.63 last 20.21) - open/close 5182 (time 1769077770.92 total 307.65 last 22.51) - open/close 5383 (time 1769077781.53 total 318.26 last 18.94) - open/close 5589 (time 1769077791.96 total 328.69 last 19.75) - open/close 5770 (time 1769077802.70 total 339.43 last 16.86) - open/close 6059 (time 1769077813.65 total 350.39 last 26.37) - open/close 6309 (time 1769077823.66 total 360.39 last 24.98) - open/close 6529 (time 1769077834.21 total 370.95 last 20.85) - open/close 6772 (time 1769077844.23 total 380.97 last 24.25) - open/close 6966 (time 1769077854.37 total 391.10 last 19.14) - open/close 7110 (time 1769077864.43 total 401.16 last 14.32) - open/close 7258 (time 1769077874.92 total 411.65 last 14.11) - open/close 7379 (time 1769077884.92 total 421.65 last 12.09) - open/close 7501 (time 1769077894.96 total 431.69 last 12.15) - open/close 7598 (time 1769077905.13 total 441.87 last 9.53) - open/close 7711 (time 1769077916.02 total 452.75 last 10.38) - open/close 7839 (time 1769077930.06 total 466.79 last 9.12) - open/close 7963 (time 1769077942.56 total 479.29 last 9.92) - open/close 8045 (time 1769077952.82 total 489.55 last 7.99) - open/close 8207 (time 1769077962.91 total 499.64 last 16.05) - open/close 8323 (time 1769077973.38 total 510.11 last 11.08) - open/close 8471 (time 1769077983.57 total 520.30 last 14.53) - open/close 8658 (time 1769077996.59 total 533.32 last 14.37) - open/close 8797 (time 1769078007.52 total 544.25 last 12.71) - open/close 9233 (time 1769078017.56 total 554.29 last 43.45) - open/close 9408 (time 1769078027.56 total 564.29 last 17.50) - open/close 9611 (time 1769078040.21 total 576.94 last 16.05) - open/close 9778 (time 1769078050.33 total 587.06 last 16.50) - open/close 9894 (time 1769078060.35 total 597.08 last 11.58) - open/close 10000 (time 1769078070.14 total 606.87 last 10.83) - open/close 10168 (time 1769078080.14 total 616.87 last 16.79) - open/close 10394 (time 1769078094.38 total 631.11 last 15.87) - open/close 10486 (time 1769078104.59 total 641.32 last 9.01) - open/close 10652 (time 1769078114.61 total 651.34 last 16.57) - open/close 10779 (time 1769078127.21 total 663.94 last 10.08) - open/close 10976 (time 1769078137.22 total 673.95 last 19.69) - open/close 11106 (time 1769078147.23 total 683.96 last 12.98) - open/close 11269 (time 1769078157.25 total 693.98 last 16.26) - open/close 11459 (time 1769078167.38 total 704.11 last 18.76) - open/close 11602 (time 1769078177.89 total 714.63 last 13.60) - open/close 11749 (time 1769078187.90 total 724.64 last 14.68) - open/close 11918 (time 1769078197.91 total 734.64 last 16.89) - open/close 12014 (time 1769078208.47 total 745.20 last 9.09) - open/close 12181 (time 1769078218.47 total 755.20 last 16.70) - open/close 12254 (time 1769078229.80 total 766.53 last 6.45) - open/close 12340 (time 1769078240.43 total 777.17 last 8.08) - open/close 12378 (time 1769078250.89 total 787.62 last 3.63) - open/close 12464 (time 1769078261.66 total 798.39 last 7.98) - open/close 12521 (time 1769078271.67 total 808.40 last 5.70) - open/close 12712 (time 1769078281.69 total 818.42 last 19.06) - open/close 12869 (time 1769078292.72 total 829.46 last 14.22) - open/close 12933 (time 1769078303.36 total 840.09 last 6.02) - open/close 13095 (time 1769078313.83 total 850.56 last 15.47) - open/close 13228 (time 1769078323.88 total 860.61 last 13.24) - open/close 13295 (time 1769078334.08 total 870.81 last 6.57) - open/close 13482 (time 1769078344.09 total 880.83 last 18.68) - open/close 13651 (time 1769078354.13 total 890.86 last 16.84) - open/close 13813 (time 1769078364.95 total 901.69 last 14.97) - open/close 14019 (time 1769078375.35 total 912.08 last 19.82) - open/close 14249 (time 1769078385.62 total 922.35 last 22.40) - open/close 14576 (time 1769078396.15 total 932.89 last 31.03) - open/close 14703 (time 1769078406.77 total 943.50 last 11.97) - open/close 14903 (time 1769078416.81 total 953.54 last 19.91) - open/close 15057 (time 1769078427.08 total 963.82 last 14.99) - open/close 15317 (time 1769078437.12 total 973.85 last 25.92) - open/close 15410 (time 1769078447.13 total 983.86 last 9.29) - open/close 15537 (time 1769078457.31 total 994.04 last 12.48) - open/close 15657 (time 1769078467.57 total 1004.31 last 11.69) - open/close 15931 (time 1769078477.58 total 1014.31 last 27.39) - open/close 15999 (time 1769078487.58 total 1024.31 last 6.80) - open/close 16054 (time 1769078498.63 total 1035.36 last 4.98) - open/close 16145 (time 1769078509.54 total 1046.28 last 8.34) - open/close 16319 (time 1769078520.05 total 1056.79 last 16.55) - open/close 16523 (time 1769078531.82 total 1068.56 last 17.33) - open/close 16782 (time 1769078542.74 total 1079.47 last 23.73) - open/close 16890 (time 1769078552.77 total 1089.50 last 10.76) - open/close 17134 (time 1769078562.77 total 1099.50 last 24.40) - open/close 17422 (time 1769078573.42 total 1110.15 last 27.04) - open/close 17619 (time 1769078583.43 total 1120.16 last 19.69) - open/close 18063 (time 1769078593.45 total 1130.19 last 44.27) - open/close 18542 (time 1769078603.46 total 1140.19 last 47.88) - open/close 19051 (time 1769078613.47 total 1150.20 last 50.85) - open/close 19678 (time 1769078623.47 total 1160.20 last 62.68) - open/close 20000 (time 1769078629.28 total 1166.01 last 55.45) - open/close 20591 (time 1769078639.28 total 1176.01 last 59.07) - open/close 21280 (time 1769078649.30 total 1186.04 last 68.74) - open/close 21885 (time 1769078659.31 total 1196.04 last 60.49) - open/close 22471 (time 1769078669.32 total 1206.05 last 58.53) - open/close 22985 (time 1769078679.33 total 1216.06 last 51.35) - open/close 23788 (time 1769078689.34 total 1226.07 last 80.25) - open/close 24500 (time 1769078699.35 total 1236.08 last 71.07) - open/close 25060 (time 1769078709.38 total 1246.11 last 55.86) - open/close 25643 (time 1769078719.38 total 1256.11 last 58.29) - open/close 26260 (time 1769078729.39 total 1266.12 last 61.65) - open/close 26811 (time 1769078739.41 total 1276.14 last 54.97) - open/close 27448 (time 1769078749.43 total 1286.16 last 63.58) - open/close 28144 (time 1769078759.43 total 1296.16 last 69.59) - open/close 28859 (time 1769078769.44 total 1306.17 last 71.47) - open/close 29543 (time 1769078779.44 total 1316.18 last 68.34) - open/close 30000 (time 1769078785.72 total 1322.45 last 72.88) - open/close 30678 (time 1769078795.72 total 1332.45 last 67.79) - open/close 31272 (time 1769078805.73 total 1342.46 last 59.35) - open/close 31917 (time 1769078815.73 total 1352.46 last 64.46) - open/close 32619 (time 1769078825.74 total 1362.47 last 70.17) - open/close 33226 (time 1769078835.76 total 1372.49 last 60.58) - open/close 33877 (time 1769078845.76 total 1382.50 last 65.05) - open/close 34379 (time 1769078855.78 total 1392.51 last 50.14) - open/close 34874 (time 1769078865.79 total 1402.52 last 49.44) - open/close 35478 (time 1769078875.79 total 1412.53 last 60.36) - open/close 36045 (time 1769078885.80 total 1422.53 last 56.69) - open/close 36578 (time 1769078895.80 total 1432.53 last 53.26) - open/close 37118 (time 1769078905.81 total 1442.54 last 53.98) - open/close 37642 (time 1769078915.82 total 1452.55 last 52.35) - open/close 38138 (time 1769078925.82 total 1462.55 last 49.58) - open/close 38691 (time 1769078935.83 total 1472.56 last 55.27) - open/close 39185 (time 1769078945.83 total 1482.56 last 49.39) - open/close 39737 (time 1769078955.84 total 1492.57 last 55.13) - open/close 40000 (time 1769078960.00 total 1496.73 last 63.30) - open/close 40535 (time 1769078970.00 total 1506.74 last 53.45) - open/close 41166 (time 1769078980.01 total 1516.74 last 63.07) - open/close 41802 (time 1769078990.01 total 1526.75 last 63.58) - open/close 42344 (time 1769079000.03 total 1536.76 last 54.13) - open/close 42894 (time 1769079010.07 total 1546.80 last 54.77) - open/close 43558 (time 1769079020.07 total 1556.80 last 66.39) - open/close 44234 (time 1769079030.08 total 1566.81 last 67.53) - open/close 44810 (time 1769079040.08 total 1576.82 last 57.57) - open/close 45423 (time 1769079050.09 total 1586.82 last 61.27) - open/close 46081 (time 1769079060.12 total 1596.85 last 65.59) - open/close 46686 (time 1769079070.12 total 1606.86 last 60.47) - open/close 47284 (time 1769079080.14 total 1616.87 last 59.73) - open/close 47856 (time 1769079090.15 total 1626.88 last 57.15) - open/close 48455 (time 1769079100.15 total 1636.88 last 59.88) - open/close 49131 (time 1769079110.16 total 1646.89 last 67.56) - open/close 49763 (time 1769079120.17 total 1656.90 last 63.14) total: 50000 open/close in 1660.15 seconds: 30.12 ops/second - unlinked 0 (time 1769079128 ; total 0 ; last 0) - unlinked 10000 (time 1769079282 ; total 154 ; last 154) - unlinked 20000 (time 1769079441 ; total 313 ; last 159) - unlinked 30000 (time 1769079577 ; total 449 ; last 136) - unlinked 40000 (time 1769079723 ; total 595 ; last 146) total: 50000 unlinks in 742 seconds: 67.385445 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2