== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:46:47 (1761241607) start mds service on oleg633-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing load_modules_local oleg633-server: Loading modules from /home/green/git/lustre-release/lustre oleg633-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg633-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg633-server: detected 4 online CPUs by sysfs oleg633-server: MODOPTS_LIBCFS= oleg633-server: Force libcfs to create 2 CPU partitions oleg633-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg633-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg633-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg633-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 180 (time 1761241695.19 total 10.01 last 17.99) - open/close 347 (time 1761241705.40 total 20.22 last 16.35) - open/close 610 (time 1761241716.41 total 31.23 last 23.89) - open/close 819 (time 1761241727.27 total 42.09 last 19.25) - open/close 997 (time 1761241737.27 total 52.09 last 17.79) - open/close 1220 (time 1761241748.14 total 62.95 last 20.53) - open/close 1390 (time 1761241758.31 total 73.13 last 16.71) - open/close 1599 (time 1761241768.33 total 83.14 last 20.86) - open/close 1960 (time 1761241778.35 total 93.17 last 36.02) - open/close 2157 (time 1761241788.91 total 103.73 last 18.65) - open/close 2371 (time 1761241799.40 total 114.22 last 20.40) - open/close 2537 (time 1761241809.42 total 124.24 last 16.57) - open/close 2683 (time 1761241819.95 total 134.76 last 13.87) - open/close 2850 (time 1761241830.76 total 145.57 last 15.45) - open/close 3061 (time 1761241841.44 total 156.26 last 19.74) - open/close 3278 (time 1761241851.45 total 166.27 last 21.69) - open/close 3490 (time 1761241861.46 total 176.28 last 21.18) - open/close 3687 (time 1761241871.54 total 186.36 last 19.54) - open/close 3984 (time 1761241881.55 total 196.37 last 29.67) - open/close 4258 (time 1761241891.56 total 206.38 last 27.38) - open/close 4532 (time 1761241901.79 total 216.61 last 26.78) - open/close 4769 (time 1761241911.81 total 226.62 last 23.67) - open/close 4987 (time 1761241921.89 total 236.70 last 21.63) - open/close 5189 (time 1761241932.70 total 247.52 last 18.68) - open/close 5407 (time 1761241943.35 total 258.17 last 20.46) - open/close 5580 (time 1761241953.36 total 268.18 last 17.29) - open/close 5664 (time 1761241963.38 total 278.20 last 8.38) - open/close 5885 (time 1761241974.33 total 289.15 last 20.19) - open/close 6120 (time 1761241984.99 total 299.81 last 22.03) - open/close 6323 (time 1761241995.01 total 309.82 last 20.28) - open/close 6533 (time 1761242005.18 total 319.99 last 20.65) - open/close 6796 (time 1761242015.98 total 330.80 last 24.34) - open/close 7034 (time 1761242026.55 total 341.36 last 22.53) - open/close 7290 (time 1761242036.56 total 351.38 last 25.57) - open/close 7485 (time 1761242047.11 total 361.92 last 18.49) - open/close 7719 (time 1761242057.12 total 371.94 last 23.36) - open/close 7923 (time 1761242068.01 total 382.82 last 18.74) - open/close 8143 (time 1761242080.23 total 395.04 last 18.00) - open/close 8382 (time 1761242090.23 total 405.05 last 23.89) - open/close 8535 (time 1761242100.33 total 415.15 last 15.14) - open/close 8727 (time 1761242110.35 total 425.16 last 19.18) - open/close 8952 (time 1761242120.69 total 435.51 last 21.75) - open/close 9183 (time 1761242130.70 total 445.51 last 23.08) - open/close 9402 (time 1761242141.23 total 456.05 last 20.79) - open/close 9629 (time 1761242151.54 total 466.35 last 22.02) - open/close 9836 (time 1761242162.31 total 477.12 last 19.22) - open/close 10000 (time 1761242171.25 total 486.07 last 18.33) - open/close 10194 (time 1761242181.48 total 496.30 last 18.97) - open/close 10366 (time 1761242192.48 total 507.30 last 15.63) - open/close 10632 (time 1761242202.51 total 517.33 last 26.53) - open/close 10823 (time 1761242213.37 total 528.19 last 17.58) - open/close 11068 (time 1761242224.05 total 538.86 last 22.96) - open/close 11282 (time 1761242234.47 total 549.29 last 20.52) - open/close 11462 (time 1761242244.59 total 559.40 last 17.80) - open/close 11686 (time 1761242255.62 total 570.43 last 20.31) - open/close 11890 (time 1761242265.70 total 580.52 last 20.23) - open/close 12091 (time 1761242275.70 total 590.52 last 20.10) - open/close 12345 (time 1761242286.64 total 601.46 last 23.21) - open/close 12534 (time 1761242296.65 total 611.46 last 18.90) - open/close 12750 (time 1761242306.84 total 621.66 last 21.18) - open/close 12977 (time 1761242316.91 total 631.73 last 22.55) - open/close 13225 (time 1761242326.93 total 641.75 last 24.74) - open/close 13431 (time 1761242336.94 total 651.75 last 20.59) - open/close 13617 (time 1761242347.25 total 662.06 last 18.04) - open/close 13849 (time 1761242357.81 total 672.63 last 21.97) - open/close 14058 (time 1761242368.04 total 682.86 last 20.42) - open/close 14254 (time 1761242378.05 total 692.87 last 19.58) - open/close 14437 (time 1761242388.52 total 703.34 last 17.48) - open/close 14650 (time 1761242398.53 total 713.35 last 21.29) - open/close 14873 (time 1761242409.55 total 724.36 last 20.24) - open/close 15100 (time 1761242419.94 total 734.76 last 21.83) - open/close 15361 (time 1761242429.96 total 744.78 last 26.06) - open/close 15563 (time 1761242440.12 total 754.93 last 19.89) - open/close 15734 (time 1761242450.86 total 765.68 last 15.92) - open/close 15915 (time 1761242461.55 total 776.36 last 16.93) - open/close 16123 (time 1761242471.55 total 786.37 last 20.79) - open/close 16373 (time 1761242481.56 total 796.37 last 24.99) - open/close 16646 (time 1761242492.60 total 807.41 last 24.73) - open/close 16879 (time 1761242502.76 total 817.58 last 22.93) - open/close 17100 (time 1761242513.72 total 828.54 last 20.16) - open/close 17294 (time 1761242524.07 total 838.89 last 18.75) - open/close 18096 (time 1761242534.08 total 848.90 last 80.08) - open/close 18871 (time 1761242544.09 total 858.91 last 77.46) - open/close 19730 (time 1761242554.10 total 868.92 last 85.83) - open/close 20000 (time 1761242557.01 total 871.83 last 92.67) - open/close 20814 (time 1761242567.01 total 881.83 last 81.38) - open/close 21604 (time 1761242577.03 total 891.85 last 78.88) - open/close 22332 (time 1761242587.04 total 901.85 last 72.74) - open/close 23082 (time 1761242597.05 total 911.87 last 74.91) - open/close 23663 (time 1761242607.07 total 921.88 last 58.00) - open/close 24196 (time 1761242617.07 total 931.89 last 53.26) - open/close 24893 (time 1761242627.08 total 941.90 last 69.64) - open/close 25639 (time 1761242637.09 total 951.90 last 74.58) - open/close 26494 (time 1761242647.09 total 961.91 last 85.46) - open/close 27432 (time 1761242657.09 total 971.91 last 93.77) - open/close 28189 (time 1761242667.10 total 981.92 last 75.63) - open/close 28922 (time 1761242677.11 total 991.93 last 73.20) - open/close 29789 (time 1761242687.12 total 1001.94 last 86.67) - open/close 30000 (time 1761242689.53 total 1004.35 last 87.44) - open/close 30845 (time 1761242699.54 total 1014.35 last 84.47) - open/close 31642 (time 1761242709.54 total 1024.36 last 79.64) - open/close 32443 (time 1761242719.55 total 1034.36 last 80.07) - open/close 33186 (time 1761242729.55 total 1044.37 last 74.26) - open/close 34036 (time 1761242739.56 total 1054.38 last 84.94) - open/close 34782 (time 1761242749.56 total 1064.38 last 74.59) - open/close 35577 (time 1761242759.57 total 1074.38 last 79.44) - open/close 36317 (time 1761242769.58 total 1084.39 last 73.93) - open/close 37078 (time 1761242779.58 total 1094.40 last 76.07) - open/close 37879 (time 1761242789.58 total 1104.40 last 80.09) - open/close 38665 (time 1761242799.59 total 1114.41 last 78.53) - open/close 39346 (time 1761242809.60 total 1124.41 last 68.05) - open/close 40000 (time 1761242818.52 total 1133.34 last 73.29) - open/close 40735 (time 1761242828.53 total 1143.35 last 73.43) - open/close 41537 (time 1761242838.54 total 1153.36 last 80.09) - open/close 42378 (time 1761242848.55 total 1163.37 last 84.03) - open/close 43215 (time 1761242858.56 total 1173.37 last 83.66) - open/close 44050 (time 1761242868.56 total 1183.38 last 83.44) - open/close 44911 (time 1761242878.57 total 1193.39 last 86.07) - open/close 45682 (time 1761242888.58 total 1203.40 last 77.00) - open/close 46348 (time 1761242898.59 total 1213.40 last 66.56) - open/close 47053 (time 1761242908.60 total 1223.41 last 70.43) - open/close 47804 (time 1761242918.60 total 1233.42 last 75.05) - open/close 48540 (time 1761242928.62 total 1243.44 last 73.47) - open/close 49211 (time 1761242938.63 total 1253.45 last 67.03) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1258.85 seconds: 39.43 ops/second - unlinked 0 (time 1761242947 ; total 0 ; last 0) - unlinked 10000 (time 1761243076 ; total 129 ; last 129) - unlinked 20000 (time 1761243202 ; total 255 ; last 126) - unlinked 30000 (time 1761243328 ; total 381 ; last 126) - unlinked 40000 (time 1761243456 ; total 509 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 634 seconds: 78.283913 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg633-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg633-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg633-server stop mds service on oleg633-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg633-server stop mds service on oleg633-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg633-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing unload_modules_local oleg633-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg633-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg633-client: oleg633-client: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-client: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2