== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:59:38 (1746158378) start mds service on oleg353-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg353-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 297 (time 1746158431.34 total 10.00 last 29.70) - open/close 567 (time 1746158441.56 total 20.21 last 26.43) - open/close 743 (time 1746158451.56 total 30.22 last 17.59) - open/close 889 (time 1746158461.56 total 40.22 last 14.60) - open/close 1072 (time 1746158471.88 total 50.54 last 17.73) - open/close 1244 (time 1746158482.15 total 60.81 last 16.75) - open/close 1389 (time 1746158492.18 total 70.83 last 14.47) - open/close 1589 (time 1746158502.50 total 81.16 last 19.37) - open/close 1982 (time 1746158513.28 total 91.94 last 36.46) - open/close 2423 (time 1746158523.39 total 102.04 last 43.64) - open/close 2650 (time 1746158535.70 total 114.36 last 18.44) - open/close 3017 (time 1746158546.28 total 124.94 last 34.68) - open/close 3378 (time 1746158557.26 total 135.92 last 32.87) - open/close 3577 (time 1746158567.47 total 146.13 last 19.49) - open/close 3999 (time 1746158577.65 total 156.31 last 41.45) - open/close 4320 (time 1746158588.45 total 167.11 last 29.72) - open/close 4597 (time 1746158598.46 total 177.12 last 27.69) - open/close 4772 (time 1746158608.72 total 187.38 last 17.05) - open/close 4905 (time 1746158618.73 total 197.38 last 13.29) - open/close 5127 (time 1746158629.62 total 208.28 last 20.38) - open/close 5387 (time 1746158639.74 total 218.40 last 25.69) - open/close 5609 (time 1746158649.74 total 228.40 last 22.19) - open/close 5826 (time 1746158660.50 total 239.16 last 20.17) - open/close 6016 (time 1746158670.88 total 249.54 last 18.30) - open/close 6264 (time 1746158680.88 total 259.54 last 24.79) - open/close 6540 (time 1746158691.40 total 270.06 last 26.24) - open/close 6849 (time 1746158701.41 total 280.07 last 30.87) - open/close 7073 (time 1746158712.33 total 290.99 last 20.50) - open/close 7324 (time 1746158722.34 total 301.00 last 25.08) - open/close 7448 (time 1746158732.34 total 311.00 last 12.40) - open/close 7690 (time 1746158742.34 total 321.00 last 24.20) - open/close 7904 (time 1746158752.40 total 331.06 last 21.27) - open/close 8222 (time 1746158762.83 total 341.49 last 30.48) - open/close 8468 (time 1746158773.22 total 351.88 last 23.69) - open/close 8638 (time 1746158783.72 total 362.38 last 16.19) - open/close 8918 (time 1746158793.72 total 372.38 last 28.00) - open/close 10000 (time 1746158800.60 total 379.26 last 157.29) - open/close 11667 (time 1746158810.60 total 389.26 last 166.64) - open/close 13859 (time 1746158820.61 total 399.27 last 219.15) - open/close 15836 (time 1746158830.61 total 409.27 last 197.64) - open/close 17336 (time 1746158840.61 total 419.27 last 149.95) - open/close 19124 (time 1746158850.62 total 429.28 last 178.71) - open/close 20000 (time 1746158855.38 total 434.04 last 183.77) - open/close 21887 (time 1746158865.39 total 444.04 last 188.69) - open/close 23366 (time 1746158875.39 total 454.05 last 147.87) - open/close 24925 (time 1746158885.39 total 464.05 last 155.88) - open/close 26477 (time 1746158895.39 total 474.05 last 155.16) - open/close 27845 (time 1746158905.39 total 484.05 last 136.76) - open/close 29049 (time 1746158915.40 total 494.05 last 120.38) - open/close 30000 (time 1746158923.00 total 501.65 last 125.13) - open/close 31570 (time 1746158933.01 total 511.67 last 156.83) - open/close 32925 (time 1746158943.01 total 521.67 last 135.47) - open/close 34375 (time 1746158953.02 total 531.68 last 144.77) - open/close 35822 (time 1746158963.03 total 541.69 last 144.60) - open/close 37047 (time 1746158973.04 total 551.69 last 122.45) - open/close 38227 (time 1746158983.04 total 561.70 last 117.92) - open/close 39335 (time 1746158993.04 total 571.70 last 110.79) - open/close 40000 (time 1746158999.37 total 578.03 last 105.07) - open/close 41210 (time 1746159009.38 total 588.04 last 120.95) - open/close 42490 (time 1746159019.38 total 598.04 last 127.90) - open/close 43633 (time 1746159029.40 total 608.06 last 114.14) - open/close 44830 (time 1746159039.40 total 618.06 last 119.67) - open/close 46064 (time 1746159049.41 total 628.07 last 123.30) - open/close 47266 (time 1746159059.41 total 638.07 last 120.15) - open/close 48541 (time 1746159069.42 total 648.08 last 127.39) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 656.24 seconds: 75.63 ops/second - unlinked 0 (time 1746159079 ; total 0 ; last 0) - unlinked 10000 (time 1746159134 ; total 55 ; last 55) - unlinked 20000 (time 1746159177 ; total 98 ; last 43) - unlinked 30000 (time 1746159221 ; total 142 ; last 44) - unlinked 40000 (time 1746159265 ; total 186 ; last 44) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 225 seconds: 220.586670 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg353-server unloading modules via unload_modules_local on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2