== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:38:54 (1786653534) start mds service on oleg216-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' llite/lustre options: 'enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg216-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg216-server: llite/lustre options: 'enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg216-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg216-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 187 (time 1786653667.10 total 10.01 last 18.69) - open/close 312 (time 1786653677.10 total 20.01 last 12.50) - open/close 366 (time 1786653687.25 total 30.16 last 5.32) - open/close 490 (time 1786653697.64 total 40.55 last 11.94) - open/close 628 (time 1786653707.64 total 50.55 last 13.80) - open/close 775 (time 1786653718.11 total 61.01 last 14.05) - open/close 861 (time 1786653728.15 total 71.06 last 8.56) - open/close 1061 (time 1786653738.16 total 81.07 last 19.98) - open/close 1184 (time 1786653748.16 total 91.07 last 12.30) - open/close 1334 (time 1786653758.75 total 101.66 last 14.17) - open/close 1382 (time 1786653769.50 total 112.41 last 4.47) - open/close 1546 (time 1786653779.51 total 122.42 last 16.39) - open/close 1679 (time 1786653789.66 total 132.57 last 13.10) - open/close 1866 (time 1786653800.49 total 143.40 last 17.26) - open/close 2031 (time 1786653811.17 total 154.08 last 15.45) - open/close 2085 (time 1786653821.91 total 164.82 last 5.03) - open/close 2206 (time 1786653831.95 total 174.86 last 12.06) - open/close 2309 (time 1786653842.87 total 185.78 last 9.43) - open/close 2421 (time 1786653853.04 total 195.95 last 11.01) - open/close 2547 (time 1786653863.64 total 206.55 last 11.88) - open/close 2679 (time 1786653873.88 total 216.79 last 12.89) - open/close 2806 (time 1786653883.89 total 226.80 last 12.69) - open/close 3039 (time 1786653894.87 total 237.78 last 21.21) - open/close 3236 (time 1786653904.88 total 247.79 last 19.69) - open/close 3372 (time 1786653915.23 total 258.13 last 13.15) - open/close 3512 (time 1786653926.07 total 268.98 last 12.91) - open/close 3630 (time 1786653936.09 total 279.00 last 11.77) - open/close 3813 (time 1786653946.09 total 289.00 last 18.30) - open/close 3968 (time 1786653956.10 total 299.01 last 15.48) - open/close 4156 (time 1786653966.12 total 309.03 last 18.78) - open/close 4379 (time 1786653977.76 total 320.67 last 19.15) - open/close 4592 (time 1786653987.78 total 330.69 last 21.26) - open/close 4738 (time 1786654000.36 total 343.27 last 11.61) - open/close 4867 (time 1786654010.79 total 353.70 last 12.36) - open/close 4951 (time 1786654021.73 total 364.64 last 7.68) - open/close 5148 (time 1786654031.76 total 374.66 last 19.65) - open/close 5268 (time 1786654041.90 total 384.81 last 11.83) - open/close 5401 (time 1786654052.59 total 395.50 last 12.44) - open/close 5577 (time 1786654063.53 total 406.44 last 16.08) - open/close 5823 (time 1786654074.53 total 417.44 last 22.37) - open/close 6025 (time 1786654085.09 total 428.00 last 19.13) - open/close 6085 (time 1786654095.09 total 438.00 last 6.00) - open/close 6206 (time 1786654105.25 total 448.16 last 11.91) - open/close 6276 (time 1786654115.26 total 458.17 last 7.00) - open/close 6422 (time 1786654125.86 total 468.77 last 13.77) - open/close 6580 (time 1786654136.16 total 479.07 last 15.34) - open/close 6659 (time 1786654146.59 total 489.50 last 7.58) - open/close 6725 (time 1786654156.60 total 499.51 last 6.60) - open/close 6836 (time 1786654167.35 total 510.26 last 10.32) - open/close 6987 (time 1786654177.37 total 520.28 last 15.07) - open/close 7057 (time 1786654188.33 total 531.24 last 6.39) - open/close 7110 (time 1786654198.95 total 541.85 last 4.99) - open/close 7226 (time 1786654209.06 total 551.96 last 11.47) - open/close 7392 (time 1786654222.82 total 565.73 last 12.06) - open/close 7482 (time 1786654233.08 total 575.99 last 8.77) - open/close 7620 (time 1786654243.11 total 586.02 last 13.77) - open/close 7784 (time 1786654254.32 total 597.23 last 14.62) - open/close 7872 (time 1786654264.66 total 607.57 last 8.51) - open/close 8029 (time 1786654274.82 total 617.73 last 15.46) - open/close 8109 (time 1786654285.28 total 628.19 last 7.65) - open/close 8228 (time 1786654297.73 total 640.64 last 9.56) - open/close 8422 (time 1786654308.24 total 651.15 last 18.45) - open/close 8522 (time 1786654318.75 total 661.66 last 9.52) - open/close 8659 (time 1786654329.39 total 672.30 last 12.88) - open/close 8807 (time 1786654339.71 total 682.62 last 14.34) - open/close 8880 (time 1786654349.72 total 692.63 last 7.30) - open/close 9038 (time 1786654359.98 total 702.89 last 15.40) - open/close 9225 (time 1786654369.98 total 712.89 last 18.70) - open/close 9358 (time 1786654380.06 total 722.97 last 13.19) - open/close 9610 (time 1786654390.07 total 732.98 last 25.18) - open/close 9783 (time 1786654401.03 total 743.94 last 15.79) - open/close 9918 (time 1786654412.13 total 755.04 last 12.16) - open/close 10000 (time 1786654414.62 total 757.53 last 32.89) - open/close 10257 (time 1786654425.55 total 768.46 last 23.52) - open/close 10481 (time 1786654435.55 total 778.46 last 22.40) - open/close 10621 (time 1786654445.56 total 788.47 last 14.00) - open/close 10719 (time 1786654455.89 total 798.80 last 9.48) - open/close 10942 (time 1786654466.46 total 809.37 last 21.10) - open/close 11017 (time 1786654476.86 total 819.77 last 7.21) - open/close 11171 (time 1786654486.87 total 829.78 last 15.38) - open/close 11417 (time 1786654496.87 total 839.78 last 24.60) - open/close 11660 (time 1786654507.22 total 850.13 last 23.47) - open/close 11798 (time 1786654517.73 total 860.64 last 13.13) - open/close 11975 (time 1786654528.60 total 871.51 last 16.29) - open/close 12165 (time 1786654539.49 total 882.40 last 17.45) - open/close 12398 (time 1786654549.50 total 892.41 last 23.28) - open/close 12502 (time 1786654561.32 total 904.23 last 8.80) - open/close 12589 (time 1786654571.49 total 914.40 last 8.55) - open/close 12816 (time 1786654581.74 total 924.65 last 22.14) - open/close 13058 (time 1786654592.73 total 935.64 last 22.02) - open/close 13250 (time 1786654608.75 total 951.66 last 11.99) - open/close 13473 (time 1786654619.27 total 962.17 last 21.20) - open/close 13760 (time 1786654629.28 total 972.19 last 28.66) - open/close 13965 (time 1786654640.16 total 983.07 last 18.84) - open/close 14154 (time 1786654650.22 total 993.13 last 18.78) - open/close 14394 (time 1786654660.22 total 1003.13 last 24.00) - open/close 14627 (time 1786654670.23 total 1013.14 last 23.29) - open/close 14824 (time 1786654680.24 total 1023.15 last 19.68) - open/close 15008 (time 1786654691.27 total 1034.18 last 16.68) - open/close 15166 (time 1786654701.31 total 1044.22 last 15.74) - open/close 15197 (time 1786654711.47 total 1054.38 last 3.05) - open/close 15430 (time 1786654722.24 total 1065.15 last 21.64) - open/close 15647 (time 1786654732.25 total 1075.16 last 21.67) - open/close 15852 (time 1786654742.25 total 1085.16 last 20.50) - open/close 16143 (time 1786654752.25 total 1095.16 last 29.09) - open/close 16811 (time 1786654762.25 total 1105.16 last 66.80) - open/close 17546 (time 1786654772.26 total 1115.17 last 73.44) - open/close 18227 (time 1786654782.27 total 1125.18 last 68.07) - open/close 19017 (time 1786654792.27 total 1135.18 last 78.94) - open/close 19930 (time 1786654802.27 total 1145.18 last 91.30) - open/close 20719 (time 1786654812.27 total 1155.18 last 78.90) - open/close 21562 (time 1786654822.28 total 1165.19 last 84.26) - open/close 22513 (time 1786654832.28 total 1175.19 last 95.08) - open/close 23343 (time 1786654842.28 total 1185.19 last 82.97) - open/close 24048 (time 1786654852.29 total 1195.20 last 70.45) - open/close 24694 (time 1786654862.30 total 1205.20 last 64.58) - open/close 25432 (time 1786654872.31 total 1215.22 last 73.68) - open/close 26227 (time 1786654882.32 total 1225.23 last 79.46) - open/close 26967 (time 1786654892.34 total 1235.25 last 73.86) - open/close 27690 (time 1786654902.34 total 1245.25 last 72.28) - open/close 28433 (time 1786654912.35 total 1255.26 last 74.23) - open/close 29222 (time 1786654922.36 total 1265.26 last 78.84) - open/close 30000 (time 1786654931.28 total 1274.19 last 87.19) - open/close 30850 (time 1786654941.28 total 1284.19 last 84.98) - open/close 31675 (time 1786654951.29 total 1294.20 last 82.39) - open/close 32378 (time 1786654961.30 total 1304.20 last 70.28) - open/close 33025 (time 1786654971.31 total 1314.22 last 64.60) - open/close 33794 (time 1786654981.32 total 1324.23 last 76.84) - open/close 34591 (time 1786654991.32 total 1334.23 last 79.69) - open/close 35441 (time 1786655001.33 total 1344.23 last 84.96) - open/close 36291 (time 1786655011.33 total 1354.23 last 85.00) - open/close 37252 (time 1786655021.33 total 1364.24 last 96.07) - open/close 38110 (time 1786655031.33 total 1374.24 last 85.78) - open/close 38915 (time 1786655041.34 total 1384.25 last 80.40) - open/close 39620 (time 1786655051.39 total 1394.30 last 70.17) - open/close 40000 (time 1786655056.42 total 1399.33 last 75.59) - open/close 40717 (time 1786655066.43 total 1409.34 last 71.64) - open/close 41559 (time 1786655076.45 total 1419.36 last 84.00) - open/close 42163 (time 1786655086.46 total 1429.37 last 60.34) - open/close 42932 (time 1786655096.46 total 1439.37 last 76.88) - open/close 43653 (time 1786655106.47 total 1449.38 last 72.06) - open/close 44417 (time 1786655116.49 total 1459.40 last 76.22) - open/close 45079 (time 1786655126.50 total 1469.41 last 66.16) - open/close 45868 (time 1786655136.50 total 1479.41 last 78.85) - open/close 46678 (time 1786655146.52 total 1489.43 last 80.90) - open/close 47336 (time 1786655156.52 total 1499.43 last 65.76) - open/close 48071 (time 1786655166.54 total 1509.45 last 73.39) - open/close 48865 (time 1786655176.54 total 1519.45 last 79.40) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1529.25 seconds: 32.45 ops/second - unlinked 0 (time 1786655191 ; total 0 ; last 0) - unlinked 10000 (time 1786655327 ; total 136 ; last 136) - unlinked 20000 (time 1786655459 ; total 268 ; last 132) - unlinked 30000 (time 1786655597 ; total 406 ; last 138) - unlinked 40000 (time 1786655729 ; total 538 ; last 132) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 671 seconds: 73.965721 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server unloading modules via unload_modules_local on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: modules unloaded. pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2