== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:40:37 (1763336437) start mds service on oleg439-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg439-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg439-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 220 (time 1763336533.71 total 10.93 last 20.13) - open/close 433 (time 1763336543.77 total 20.98 last 21.18) - open/close 655 (time 1763336553.77 total 30.99 last 22.19) - open/close 838 (time 1763336564.44 total 41.66 last 17.14) - open/close 1047 (time 1763336574.55 total 51.77 last 20.68) - open/close 1259 (time 1763336584.55 total 61.77 last 21.19) - open/close 1459 (time 1763336595.31 total 72.52 last 18.60) - open/close 1689 (time 1763336605.33 total 82.55 last 22.95) - open/close 1882 (time 1763336615.63 total 92.85 last 18.73) - open/close 2054 (time 1763336626.59 total 103.81 last 15.69) - open/close 2244 (time 1763336637.15 total 114.37 last 17.99) - open/close 2474 (time 1763336647.16 total 124.38 last 22.98) - open/close 2694 (time 1763336657.31 total 134.53 last 21.68) - open/close 2974 (time 1763336667.67 total 144.89 last 27.02) - open/close 3198 (time 1763336677.68 total 154.90 last 22.38) - open/close 3415 (time 1763336687.69 total 164.90 last 21.69) - open/close 3594 (time 1763336698.14 total 175.36 last 17.12) - open/close 3796 (time 1763336708.15 total 185.37 last 20.17) - open/close 3985 (time 1763336718.42 total 195.63 last 18.42) - open/close 4197 (time 1763336728.55 total 205.76 last 20.93) - open/close 4459 (time 1763336739.54 total 216.76 last 23.83) - open/close 4646 (time 1763336749.55 total 226.77 last 18.68) - open/close 4895 (time 1763336760.54 total 237.75 last 22.67) - open/close 5118 (time 1763336770.90 total 248.12 last 21.52) - open/close 5352 (time 1763336781.73 total 258.95 last 21.61) - open/close 5562 (time 1763336792.79 total 270.01 last 18.99) - open/close 5781 (time 1763336803.68 total 280.90 last 20.11) - open/close 5996 (time 1763336814.31 total 291.53 last 20.22) - open/close 6219 (time 1763336824.33 total 301.55 last 22.26) - open/close 6390 (time 1763336834.51 total 311.73 last 16.79) - open/close 6610 (time 1763336844.64 total 321.85 last 21.74) - open/close 6829 (time 1763336854.64 total 331.86 last 21.89) - open/close 7044 (time 1763336864.65 total 341.86 last 21.48) - open/close 7252 (time 1763336875.16 total 352.38 last 19.78) - open/close 7444 (time 1763336885.19 total 362.40 last 19.16) - open/close 7646 (time 1763336895.47 total 372.68 last 19.65) - open/close 7883 (time 1763336906.52 total 383.74 last 21.44) - open/close 8142 (time 1763336917.34 total 394.56 last 23.93) - open/close 8353 (time 1763336927.64 total 404.85 last 20.50) - open/close 8587 (time 1763336938.36 total 415.58 last 21.81) - open/close 8835 (time 1763336948.95 total 426.17 last 23.42) - open/close 9044 (time 1763336958.96 total 436.17 last 20.89) - open/close 9222 (time 1763336972.03 total 449.25 last 13.61) - open/close 9430 (time 1763336982.55 total 459.76 last 19.79) - open/close 9542 (time 1763336992.90 total 470.12 last 10.82) - open/close 9611 (time 1763337004.71 total 481.93 last 5.84) - open/close 9836 (time 1763337014.93 total 492.15 last 22.02) - open/close 10000 (time 1763337022.32 total 499.54 last 22.19) - open/close 10226 (time 1763337033.29 total 510.51 last 20.61) - open/close 10445 (time 1763337043.48 total 520.70 last 21.49) - open/close 10721 (time 1763337054.16 total 531.38 last 25.83) - open/close 10929 (time 1763337064.18 total 541.40 last 20.76) - open/close 11183 (time 1763337075.09 total 552.31 last 23.28) - open/close 11407 (time 1763337085.10 total 562.31 last 22.40) - open/close 11635 (time 1763337095.38 total 572.59 last 22.18) - open/close 11793 (time 1763337105.40 total 582.61 last 15.77) - open/close 12022 (time 1763337115.95 total 593.17 last 21.70) - open/close 12256 (time 1763337126.24 total 603.46 last 22.75) - open/close 12487 (time 1763337136.60 total 613.82 last 22.28) - open/close 12742 (time 1763337147.10 total 624.32 last 24.29) - open/close 12963 (time 1763337157.11 total 634.33 last 22.08) - open/close 13185 (time 1763337167.57 total 644.79 last 21.23) - open/close 13424 (time 1763337178.33 total 655.54 last 22.22) - open/close 13646 (time 1763337188.57 total 665.79 last 21.67) - open/close 13864 (time 1763337198.75 total 675.97 last 21.42) - open/close 14084 (time 1763337208.79 total 686.00 last 21.92) - open/close 14306 (time 1763337218.80 total 696.02 last 22.17) - open/close 14515 (time 1763337229.78 total 707.00 last 19.03) - open/close 14736 (time 1763337239.79 total 717.01 last 22.07) - open/close 14901 (time 1763337250.15 total 727.36 last 15.94) - open/close 15144 (time 1763337260.37 total 737.59 last 23.77) - open/close 15360 (time 1763337271.08 total 748.30 last 20.17) - open/close 15572 (time 1763337281.09 total 758.31 last 21.17) - open/close 15798 (time 1763337291.10 total 768.31 last 22.59) - open/close 16052 (time 1763337301.74 total 778.95 last 23.88) - open/close 16293 (time 1763337312.32 total 789.54 last 22.77) - open/close 16528 (time 1763337322.38 total 799.60 last 23.35) - open/close 16689 (time 1763337333.18 total 810.40 last 14.91) - open/close 16848 (time 1763337343.65 total 820.87 last 15.19) - open/close 17106 (time 1763337353.66 total 830.87 last 25.79) - open/close 17368 (time 1763337364.12 total 841.34 last 25.03) - open/close 17819 (time 1763337374.13 total 851.35 last 45.08) - open/close 18530 (time 1763337384.15 total 861.36 last 70.98) - open/close 19343 (time 1763337394.15 total 871.37 last 81.28) - open/close 20000 (time 1763337403.68 total 880.90 last 68.89) - open/close 20759 (time 1763337413.69 total 890.91 last 75.85) - open/close 21555 (time 1763337423.70 total 900.92 last 79.55) - open/close 22252 (time 1763337433.70 total 910.92 last 69.66) - open/close 22859 (time 1763337443.72 total 920.94 last 60.59) - open/close 23541 (time 1763337453.73 total 930.95 last 68.14) - open/close 24226 (time 1763337463.73 total 940.95 last 68.48) - open/close 24959 (time 1763337473.74 total 950.95 last 73.28) - open/close 25704 (time 1763337483.74 total 960.96 last 74.47) - open/close 26468 (time 1763337493.75 total 970.97 last 76.29) - open/close 27219 (time 1763337503.77 total 980.98 last 74.99) - open/close 27949 (time 1763337513.77 total 990.99 last 72.99) - open/close 28639 (time 1763337523.77 total 1000.99 last 68.98) - open/close 29324 (time 1763337533.78 total 1011.00 last 68.41) - open/close 29993 (time 1763337543.80 total 1021.01 last 66.83) - open/close 30676 (time 1763337553.80 total 1031.02 last 68.27) - open/close 31425 (time 1763337563.80 total 1041.02 last 74.90) - open/close 32248 (time 1763337573.80 total 1051.02 last 82.29) - open/close 32940 (time 1763337583.80 total 1061.02 last 69.19) - open/close 33642 (time 1763337593.81 total 1071.03 last 70.16) - open/close 34401 (time 1763337603.82 total 1081.04 last 75.82) - open/close 35117 (time 1763337613.83 total 1091.04 last 71.56) - open/close 35814 (time 1763337623.83 total 1101.05 last 69.66) - open/close 36481 (time 1763337633.84 total 1111.06 last 66.64) - open/close 37170 (time 1763337643.86 total 1121.08 last 68.79) - open/close 37905 (time 1763337653.86 total 1131.08 last 73.46) - open/close 38592 (time 1763337663.88 total 1141.10 last 68.59) - open/close 39330 (time 1763337673.89 total 1151.11 last 73.73) - open/close 40000 (time 1763337683.88 total 1161.09 last 67.10) - open/close 40649 (time 1763337693.88 total 1171.10 last 64.86) - open/close 41356 (time 1763337703.88 total 1181.10 last 70.68) - open/close 42006 (time 1763337713.89 total 1191.11 last 64.96) - open/close 42735 (time 1763337723.90 total 1201.11 last 72.85) - open/close 43485 (time 1763337733.90 total 1211.12 last 74.96) - open/close 44189 (time 1763337743.91 total 1221.12 last 70.38) - open/close 44920 (time 1763337753.91 total 1231.13 last 73.06) - open/close 45633 (time 1763337763.92 total 1241.14 last 71.25) - open/close 46366 (time 1763337773.92 total 1251.14 last 73.29) - open/close 47082 (time 1763337783.92 total 1261.14 last 71.58) - open/close 47805 (time 1763337793.93 total 1271.15 last 72.22) - open/close 48486 (time 1763337803.94 total 1281.16 last 68.04) - open/close 49201 (time 1763337813.96 total 1291.17 last 71.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1297.83 seconds: 38.24 ops/second - unlinked 0 (time 1763337825 ; total 0 ; last 0) - unlinked 10000 (time 1763337966 ; total 141 ; last 141) - unlinked 20000 (time 1763338107 ; total 282 ; last 141) - unlinked 30000 (time 1763338246 ; total 421 ; last 139) - unlinked 40000 (time 1763338384 ; total 559 ; last 138) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 700 seconds: 70.902855 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local oleg439-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2