== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:51:35 (1718398295) Stopping clients: oleg228-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg228-client.virtnet /mnt/lustre2 (opts:-f) oleg228-server: oleg228-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: detected 4 online CPUs by sysfs oleg228-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg228-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg228-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg228-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock oleg228-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2978 (time 1718398347.63 total 10.01 last 297.58) - open/close 5278 (time 1718398357.63 total 20.01 last 229.97) - open/close 8018 (time 1718398367.63 total 30.01 last 273.95) - open/close 10000 (time 1718398376.01 total 38.39 last 236.60) - open/close 12708 (time 1718398386.01 total 48.39 last 270.77) - open/close 14947 (time 1718398396.01 total 58.39 last 223.90) - open/close 17290 (time 1718398406.01 total 68.39 last 234.25) - open/close 19721 (time 1718398416.01 total 78.39 last 243.10) - open/close 22292 (time 1718398426.01 total 88.39 last 257.05) - open/close 24840 (time 1718398436.02 total 98.40 last 254.69) - open/close 26925 (time 1718398446.02 total 108.40 last 208.45) - open/close 29223 (time 1718398456.02 total 118.40 last 229.73) - open/close 30000 (time 1718398459.36 total 121.74 last 232.77) - open/close 32754 (time 1718398469.36 total 131.74 last 275.32) - open/close 35572 (time 1718398479.37 total 141.75 last 281.71) - open/close 38378 (time 1718398489.37 total 151.75 last 280.55) - open/close 40000 (time 1718398494.65 total 157.03 last 307.02) - open/close 43089 (time 1718398504.65 total 167.03 last 308.86) - open/close 46050 (time 1718398514.66 total 177.04 last 296.04) - open/close 48707 (time 1718398524.66 total 187.04 last 265.65) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 190.78 seconds: 260.50 ops/second - /unlink 4167 (time 1718398539.53 total 10.00 last 416.69) - /unlink 9123 (time 1718398549.53 total 20.00 last 495.56) - /unlink 14199 (time 1718398559.53 total 30.00 last 507.54) - /unlink 18426 (time 1718398569.53 total 40.00 last 422.62) - /unlink 20000 (time 1718398572.92 total 43.39 last 464.26) - /unlink 24663 (time 1718398582.93 total 53.40 last 466.24) - /unlink 29230 (time 1718398592.93 total 63.40 last 456.67) - /unlink 33982 (time 1718398602.93 total 73.40 last 475.14) - /unlink 38974 (time 1718398612.93 total 83.40 last 499.09) - /unlink 40000 (time 1718398615.61 total 86.08 last 383.46) - /unlink 45057 (time 1718398625.61 total 96.08 last 505.66) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 104.69 seconds: 474.69 ops/second - open/close 2953 (time 1718398645.46 total 10.00 last 295.24) - open/close 6415 (time 1718398655.46 total 20.00 last 346.12) - open/close 9120 (time 1718398665.46 total 30.01 last 270.48) - open/close 10000 (time 1718398668.24 total 32.79 last 316.39) - open/close 13351 (time 1718398678.25 total 42.79 last 335.04) - open/close 16053 (time 1718398688.25 total 52.79 last 270.18) - open/close 18812 (time 1718398698.25 total 62.79 last 275.78) - open/close 20000 (time 1718398702.88 total 67.42 last 256.53) - open/close 22731 (time 1718398712.89 total 77.43 last 273.00) - open/close 25395 (time 1718398722.89 total 87.43 last 266.38) - open/close 28209 (time 1718398732.89 total 97.43 last 281.30) - open/close 30000 (time 1718398738.72 total 103.26 last 307.09) - open/close 32896 (time 1718398748.72 total 113.27 last 289.58) - open/close 35439 (time 1718398758.72 total 123.27 last 254.26) - open/close 37976 (time 1718398768.73 total 133.27 last 253.62) - open/close 40000 (time 1718398776.14 total 140.68 last 273.18) - open/close 42931 (time 1718398786.14 total 150.68 last 293.03) - open/close 45605 (time 1718398796.14 total 160.68 last 267.33) - open/close 48716 (time 1718398806.14 total 170.69 last 311.07) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 173.62 seconds: 286.24 ops/second - /unlink 5714 (time 1718398820.15 total 10.00 last 571.35) - /unlink 10000 (time 1718398827.75 total 17.61 last 563.50) - /unlink 15368 (time 1718398837.75 total 27.61 last 536.69) - /unlink 20000 (time 1718398845.85 total 35.71 last 572.05) - /unlink 25358 (time 1718398855.85 total 45.71 last 535.79) - /unlink 30000 (time 1718398863.66 total 53.52 last 594.35) - /unlink 35518 (time 1718398873.66 total 63.52 last 551.73) - /unlink 40000 (time 1718398880.59 total 70.45 last 646.70) - /unlink 46958 (time 1718398890.59 total 80.45 last 695.80) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 84.26 seconds: 589.81 ops/second - open/close 3757 (time 1718398905.38 total 10.00 last 375.62) - open/close 7139 (time 1718398915.39 total 20.00 last 338.17) - open/close 10000 (time 1718398924.02 total 28.64 last 331.26) - open/close 13251 (time 1718398934.02 total 38.64 last 325.06) - open/close 16397 (time 1718398944.02 total 48.64 last 314.56) - open/close 19718 (time 1718398954.03 total 58.64 last 332.05) - open/close 23676 (time 1718398964.03 total 68.65 last 395.73) - open/close 27234 (time 1718398974.03 total 78.65 last 355.77) - open/close 30000 (time 1718398982.15 total 86.77 last 340.56) - open/close 31756 (time 1718398992.15 total 96.77 last 175.53) - open/close 33025 (time 1718399002.16 total 106.77 last 126.88) - open/close 34269 (time 1718399012.16 total 116.78 last 124.35) - open/close 35554 (time 1718399022.17 total 126.78 last 128.44) - open/close 37105 (time 1718399032.17 total 136.79 last 155.05) - open/close 38751 (time 1718399042.17 total 146.79 last 164.56) - open/close 40000 (time 1718399049.51 total 154.13 last 170.15) - open/close 41562 (time 1718399059.52 total 164.13 last 156.14) - open/close 42866 (time 1718399069.52 total 174.13 last 130.40) - open/close 44107 (time 1718399079.52 total 184.13 last 124.09) - open/close 45482 (time 1718399089.52 total 194.14 last 137.45) - open/close 46998 (time 1718399099.52 total 204.14 last 151.55) - open/close 48914 (time 1718399109.52 total 214.14 last 191.57) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 218.81 seconds: 227.13 ops/second - /unlink 2664 (time 1718399125.66 total 10.00 last 266.31) - /unlink 5149 (time 1718399135.66 total 20.00 last 248.46) - /unlink 8020 (time 1718399145.66 total 30.01 last 287.06) - /unlink 10000 (time 1718399152.77 total 37.12 last 278.44) - /unlink 12711 (time 1718399162.77 total 47.12 last 271.04) - /unlink 16021 (time 1718399172.77 total 57.12 last 330.99) - /unlink 18720 (time 1718399182.78 total 67.12 last 269.83) - /unlink 20000 (time 1718399186.17 total 70.51 last 377.70) - /unlink 23314 (time 1718399196.17 total 80.51 last 331.34) - /unlink 26228 (time 1718399206.17 total 90.51 last 291.37) - /unlink 28882 (time 1718399216.17 total 100.52 last 265.35) - /unlink 30000 (time 1718399221.71 total 106.06 last 201.62) - /unlink 32441 (time 1718399231.72 total 116.06 last 244.05) - /unlink 35001 (time 1718399241.72 total 126.07 last 255.89) - /unlink 37321 (time 1718399251.73 total 136.07 last 231.90) - /unlink 40000 (time 1718399261.05 total 145.39 last 287.42) - /unlink 42540 (time 1718399271.05 total 155.39 last 254.00) - /unlink 44921 (time 1718399281.05 total 165.39 last 238.05) - /unlink 47497 (time 1718399291.05 total 175.40 last 257.56) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 184.41 seconds: 269.50 ops/second - open/close 1386 (time 1718399312.01 total 10.00 last 138.55) - open/close 2744 (time 1718399322.01 total 20.01 last 135.77) - open/close 4186 (time 1718399332.02 total 30.01 last 144.13) - open/close 5772 (time 1718399342.02 total 40.02 last 158.48) - open/close 7361 (time 1718399352.02 total 50.02 last 158.90) - open/close 9378 (time 1718399362.03 total 60.03 last 201.57) - open/close 10000 (time 1718399364.82 total 62.81 last 223.18) - open/close 12033 (time 1718399374.82 total 72.82 last 203.21) - open/close 13852 (time 1718399384.83 total 82.82 last 181.79) - open/close 15556 (time 1718399394.83 total 92.82 last 170.39) - open/close 17158 (time 1718399404.83 total 102.82 last 160.18) - open/close 18687 (time 1718399414.83 total 112.83 last 152.85) - open/close 20000 (time 1718399424.36 total 122.35 last 137.87) - open/close 21323 (time 1718399434.36 total 132.36 last 132.25) - open/close 22717 (time 1718399444.37 total 142.36 last 139.28) - open/close 24112 (time 1718399454.37 total 152.37 last 139.47) - open/close 25467 (time 1718399464.38 total 162.37 last 135.42) - open/close 26832 (time 1718399474.38 total 172.37 last 136.47) - open/close 28254 (time 1718399484.38 total 182.38 last 142.17) - open/close 29633 (time 1718399494.39 total 192.38 last 137.84) - open/close 30000 (time 1718399496.48 total 194.47 last 175.64) - open/close 31555 (time 1718399506.48 total 204.47 last 155.49) - open/close 33170 (time 1718399516.48 total 214.47 last 161.48) - open/close 35166 (time 1718399526.48 total 224.48 last 199.52) - open/close 37348 (time 1718399536.49 total 234.48 last 218.10) - open/close 39262 (time 1718399546.49 total 244.49 last 191.29) - open/close 40000 (time 1718399550.22 total 248.21 last 198.20) - open/close 41689 (time 1718399560.22 total 258.21 last 168.88) - open/close 43427 (time 1718399570.22 total 268.22 last 173.72) - open/close 45242 (time 1718399580.23 total 278.22 last 181.42) - open/close 47114 (time 1718399590.23 total 288.22 last 187.20) open(/mnt/lustre/d106.conf-sanity/f-48488) error: No space left on device total: 48488 open/close in 295.88 seconds: 163.88 ops/second - /unlink 3020 (time 1718399609.40 total 10.00 last 301.90) - /unlink 6115 (time 1718399619.40 total 20.00 last 309.50) - /unlink 9554 (time 1718399629.40 total 30.00 last 343.90) - /unlink 13766 (time 1718399639.40 total 40.00 last 421.20) - /unlink 18283 (time 1718399649.40 total 50.01 last 451.58) - /unlink 20000 (time 1718399652.87 total 53.47 last 495.18) - /unlink 26806 (time 1718399662.87 total 63.47 last 680.54) - /unlink 30000 (time 1718399666.79 total 67.39 last 816.08) - /unlink 37791 (time 1718399676.79 total 77.39 last 779.01) - /unlink 40000 (time 1718399679.59 total 80.19 last 789.32) - /unlink 47616 (time 1718399689.59 total 90.19 last 761.53) unlink(/mnt/lustre/d106.conf-sanity/f-48488) error: No such file or directory total: 48488 /unlink in 91.08 seconds: 532.35 ops/second - open/close 4624 (time 1718399701.36 total 10.00 last 462.28) - open/close 8841 (time 1718399711.36 total 20.01 last 421.57) - open/close 10000 (time 1718399714.20 total 22.84 last 408.26) - open/close 14184 (time 1718399724.21 total 32.86 last 417.86) - open/close 18518 (time 1718399734.21 total 42.86 last 433.36) - open/close 20000 (time 1718399737.82 total 46.46 last 411.30) - open/close 24817 (time 1718399747.82 total 56.46 last 481.69) - open/close 29691 (time 1718399757.82 total 66.46 last 487.38) - open/close 34527 (time 1718399767.82 total 76.46 last 483.59) - open/close 38784 (time 1718399777.82 total 86.46 last 425.63) - open/close 40000 (time 1718399780.34 total 88.99 last 481.89) - open/close 44159 (time 1718399790.34 total 98.99 last 415.87) open(/mnt/lustre/d106.conf-sanity/f-48488) error: No space left on device total: 48488 open/close in 108.10 seconds: 448.56 ops/second - /unlink 8416 (time 1718399810.36 total 10.00 last 841.55) - /unlink 18465 (time 1718399820.36 total 20.00 last 1004.84) - /unlink 28261 (time 1718399830.36 total 30.00 last 979.53) - /unlink 38323 (time 1718399840.36 total 40.00 last 1006.14) unlink(/mnt/lustre/d106.conf-sanity/f-48488) error: No such file or directory total: 48488 /unlink in 49.40 seconds: 981.59 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg228-server LNET ready to unload unloading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: LNET ready to unload modules unloaded. pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2