== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:47:09 (1787863629) Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg223-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg223-server oleg223-server: oleg223-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg223-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg223-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 617 (time 1787863849.31 total 10.00 last 61.69) - open/close 1195 (time 1787863859.33 total 20.03 last 57.66) - open/close 1785 (time 1787863869.34 total 30.03 last 58.97) - open/close 2480 (time 1787863879.36 total 40.05 last 69.36) - open/close 3087 (time 1787863889.37 total 50.07 last 60.61) - open/close 3656 (time 1787863899.38 total 60.07 last 56.87) - open/close 4353 (time 1787863909.38 total 70.07 last 69.68) total: 4500 open/close in 72.26 seconds: 62.28 ops/second - /unlink 1387 (time 1787863926.93 total 10.00 last 138.66) - /unlink 2563 (time 1787863936.95 total 20.02 last 117.42) - /unlink 4035 (time 1787863946.95 total 30.02 last 147.20) total: 4500 /unlink in 33.34 seconds: 134.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 683 (time 1787863965.70 total 10.01 last 68.22) - open/close 1462 (time 1787863975.70 total 20.02 last 77.87) - open/close 2245 (time 1787863985.70 total 30.02 last 78.30) - open/close 3099 (time 1787863995.70 total 40.02 last 85.40) - open/close 3864 (time 1787864005.71 total 50.02 last 76.44) total: 4500 open/close in 59.87 seconds: 75.17 ops/second - /unlink 1116 (time 1787864032.01 total 10.01 last 111.47) - /unlink 2425 (time 1787864042.02 total 20.02 last 130.83) - /unlink 3710 (time 1787864052.03 total 30.02 last 128.41) total: 4500 /unlink in 36.68 seconds: 122.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 619 (time 1787864074.01 total 10.01 last 61.86) - open/close 1328 (time 1787864084.02 total 20.02 last 70.84) - open/close 2106 (time 1787864094.02 total 30.02 last 77.79) - open/close 2791 (time 1787864104.02 total 40.02 last 68.48) - open/close 3446 (time 1787864114.03 total 50.03 last 65.47) - open/close 4068 (time 1787864124.03 total 60.03 last 62.18) total: 4500 open/close in 65.73 seconds: 68.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1310 (time 1787864144.25 total 10.01 last 130.93) - /unlink 2540 (time 1787864154.25 total 20.01 last 122.96) - /unlink 3514 (time 1787864164.25 total 30.01 last 97.40) total: 4500 /unlink in 37.43 seconds: 120.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 614 (time 1787864189.33 total 10.01 last 61.35) - open/close 1273 (time 1787864199.35 total 20.02 last 65.82) - open/close 1962 (time 1787864209.35 total 30.02 last 68.89) - open/close 2536 (time 1787864219.35 total 40.03 last 57.36) - open/close 3234 (time 1787864229.36 total 50.03 last 69.76) - open/close 3914 (time 1787864239.38 total 60.05 last 67.88) total: 4500 open/close in 69.10 seconds: 65.12 ops/second - /unlink 1402 (time 1787864264.73 total 10.01 last 140.13) - /unlink 2839 (time 1787864274.74 total 20.01 last 143.64) - /unlink 4086 (time 1787864284.74 total 30.01 last 124.69) total: 4500 /unlink in 33.88 seconds: 132.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 656 (time 1787864304.10 total 10.00 last 65.57) - open/close 1349 (time 1787864314.10 total 20.01 last 69.25) - open/close 2004 (time 1787864324.12 total 30.02 last 65.41) - open/close 2615 (time 1787864334.13 total 40.04 last 61.02) - open/close 3238 (time 1787864344.14 total 50.04 last 62.26) - open/close 3868 (time 1787864354.14 total 60.05 last 63.00) total: 4500 open/close in 68.23 seconds: 65.95 ops/second - /unlink 1300 (time 1787864378.40 total 10.01 last 129.85) - /unlink 2370 (time 1787864388.40 total 20.02 last 106.95) - /unlink 3553 (time 1787864398.40 total 30.02 last 118.28) total: 4500 /unlink in 37.34 seconds: 120.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 788 (time 1787864421.03 total 10.01 last 78.74) - open/close 1632 (time 1787864431.03 total 20.01 last 84.38) - open/close 2348 (time 1787864441.04 total 30.02 last 71.56) - open/close 3037 (time 1787864451.04 total 40.02 last 68.88) - open/close 3714 (time 1787864461.05 total 50.03 last 67.63) - open/close 4358 (time 1787864471.06 total 60.04 last 64.35) total: 4500 open/close in 62.75 seconds: 71.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1207 (time 1787864488.83 total 10.00 last 120.68) - /unlink 2476 (time 1787864498.84 total 20.02 last 126.71) - /unlink 3751 (time 1787864508.85 total 30.02 last 127.45) total: 4500 /unlink in 35.59 seconds: 126.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 713 (time 1787864530.13 total 10.01 last 71.26) - open/close 1354 (time 1787864540.14 total 20.01 last 64.05) - open/close 1909 (time 1787864550.14 total 30.02 last 55.46) - open/close 2519 (time 1787864560.15 total 40.03 last 60.97) - open/close 3254 (time 1787864570.15 total 50.03 last 73.49) - open/close 4040 (time 1787864580.16 total 60.04 last 78.54) total: 4500 open/close in 65.70 seconds: 68.50 ops/second - /unlink 1203 (time 1787864600.96 total 10.00 last 120.29) - /unlink 2508 (time 1787864610.97 total 20.01 last 130.36) - /unlink 3866 (time 1787864620.97 total 30.01 last 135.79) total: 4500 /unlink in 34.28 seconds: 131.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 724 (time 1787864640.70 total 10.00 last 72.37) - open/close 1421 (time 1787864650.70 total 20.00 last 69.70) - open/close 2150 (time 1787864660.71 total 30.02 last 72.80) - open/close 2824 (time 1787864670.72 total 40.03 last 67.31) - open/close 3471 (time 1787864680.77 total 50.08 last 64.41) - open/close 4060 (time 1787864690.80 total 60.10 last 58.73) total: 4500 open/close in 66.69 seconds: 67.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1221 (time 1787864712.32 total 10.00 last 122.07) - /unlink 2569 (time 1787864722.32 total 20.01 last 134.76) - /unlink 3808 (time 1787864732.33 total 30.01 last 123.85) total: 4500 /unlink in 35.12 seconds: 128.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 870 (time 1787864751.81 total 10.01 last 86.96) - open/close 1717 (time 1787864761.81 total 20.01 last 84.67) - open/close 2508 (time 1787864771.82 total 30.02 last 79.02) - open/close 3385 (time 1787864781.83 total 40.03 last 87.61) - open/close 4176 (time 1787864791.85 total 50.04 last 79.00) total: 4500 open/close in 53.41 seconds: 84.25 ops/second - /unlink 1439 (time 1787864808.95 total 10.00 last 143.86) - /unlink 3080 (time 1787864818.96 total 20.01 last 164.05) total: 4500 /unlink in 28.32 seconds: 158.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 860 (time 1787864842.13 total 10.01 last 85.95) - open/close 1782 (time 1787864852.14 total 20.01 last 92.14) - open/close 2482 (time 1787864862.14 total 30.02 last 69.95) - open/close 3160 (time 1787864872.15 total 40.02 last 67.77) - open/close 3831 (time 1787864882.15 total 50.03 last 67.06) total: 4500 open/close in 59.33 seconds: 75.84 ops/second - /unlink 1416 (time 1787864906.20 total 10.00 last 141.58) - /unlink 2506 (time 1787864916.20 total 20.00 last 108.98) - /unlink 3616 (time 1787864926.21 total 30.01 last 110.98) total: 4500 /unlink in 36.31 seconds: 123.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 688 (time 1787864947.39 total 10.01 last 68.76) - open/close 1464 (time 1787864957.41 total 20.03 last 77.44) - open/close 2051 (time 1787864967.42 total 30.03 last 58.67) - open/close 2551 (time 1787864977.43 total 40.04 last 49.94) - open/close 3262 (time 1787864987.43 total 50.05 last 71.06) - open/close 3977 (time 1787864997.43 total 60.05 last 71.48) total: 4500 open/close in 67.47 seconds: 66.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1515 (time 1787865019.28 total 10.01 last 151.42) - /unlink 3127 (time 1787865029.28 total 20.01 last 161.18) total: 4500 /unlink in 28.71 seconds: 156.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 875 (time 1787865052.66 total 10.04 last 87.18) - open/close 1830 (time 1787865062.66 total 20.04 last 95.43) - open/close 2565 (time 1787865072.67 total 30.04 last 73.50) - open/close 3428 (time 1787865082.69 total 40.06 last 86.13) - open/close 4221 (time 1787865092.70 total 50.08 last 79.20) total: 4500 open/close in 53.16 seconds: 84.66 ops/second - /unlink 1651 (time 1787865109.68 total 10.00 last 165.10) - /unlink 2996 (time 1787865119.68 total 20.00 last 134.49) total: 4500 /unlink in 29.31 seconds: 153.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 861 (time 1787865143.81 total 10.00 last 86.07) - open/close 1776 (time 1787865153.84 total 20.04 last 91.21) - open/close 2596 (time 1787865163.84 total 30.04 last 81.97) - open/close 3574 (time 1787865173.85 total 40.04 last 97.78) total: 4500 open/close in 49.90 seconds: 90.18 ops/second - /unlink 1645 (time 1787865197.61 total 10.01 last 164.39) - /unlink 3315 (time 1787865207.62 total 20.02 last 166.85) total: 4500 /unlink in 28.19 seconds: 159.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 934 (time 1787865229.83 total 10.00 last 93.39) - open/close 1855 (time 1787865239.84 total 20.01 last 92.02) - open/close 2790 (time 1787865249.86 total 30.03 last 93.31) - open/close 3604 (time 1787865259.86 total 40.03 last 81.37) - open/close 4438 (time 1787865269.88 total 50.05 last 83.27) total: 4500 open/close in 51.19 seconds: 87.91 ops/second - /unlink 1803 (time 1787865285.27 total 10.00 last 180.28) - /unlink 3372 (time 1787865295.27 total 20.00 last 156.87) total: 4500 /unlink in 28.27 seconds: 159.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 677 (time 1787865319.58 total 10.00 last 67.67) - open/close 1347 (time 1787865329.58 total 20.01 last 67.00) - open/close 2054 (time 1787865339.58 total 30.01 last 70.69) - open/close 2805 (time 1787865349.60 total 40.02 last 74.97) - open/close 3522 (time 1787865359.61 total 50.03 last 71.67) - open/close 4380 (time 1787865369.61 total 60.03 last 85.80) total: 4500 open/close in 61.30 seconds: 73.41 ops/second - /unlink 1219 (time 1787865385.96 total 10.01 last 121.73) - /unlink 2746 (time 1787865395.97 total 20.02 last 152.55) - /unlink 4028 (time 1787865405.98 total 30.03 last 128.15) total: 4500 /unlink in 34.00 seconds: 132.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 639 (time 1787865425.61 total 10.01 last 63.85) - open/close 1435 (time 1787865435.62 total 20.01 last 79.55) - open/close 2240 (time 1787865445.63 total 30.02 last 80.44) - open/close 2938 (time 1787865455.64 total 40.03 last 69.72) - open/close 3753 (time 1787865465.65 total 50.04 last 81.43) total: 4500 open/close in 58.18 seconds: 77.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1518 (time 1787865488.68 total 10.00 last 151.79) - /unlink 3096 (time 1787865498.69 total 20.00 last 157.74) total: 4500 /unlink in 27.82 seconds: 161.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 707 (time 1787865521.92 total 10.02 last 70.54) - open/close 1380 (time 1787865531.95 total 20.05 last 67.13) - open/close 1993 (time 1787865541.95 total 30.05 last 61.28) - open/close 2784 (time 1787865551.96 total 40.06 last 79.03) - open/close 3604 (time 1787865561.97 total 50.07 last 81.95) - open/close 4485 (time 1787865571.97 total 60.07 last 88.09) total: 4500 open/close in 60.21 seconds: 74.74 ops/second - /unlink 1636 (time 1787865586.71 total 10.00 last 163.52) - /unlink 3179 (time 1787865596.71 total 20.01 last 154.28) total: 4500 /unlink in 28.73 seconds: 156.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 888 (time 1787865620.38 total 10.01 last 88.74) - open/close 1746 (time 1787865630.39 total 20.02 last 85.72) - open/close 2660 (time 1787865640.40 total 30.02 last 91.34) - open/close 3521 (time 1787865650.40 total 40.03 last 86.04) - open/close 4285 (time 1787865660.41 total 50.04 last 76.32) total: 4500 open/close in 53.72 seconds: 83.76 ops/second - /unlink 1565 (time 1787865678.85 total 10.01 last 156.34) - /unlink 3095 (time 1787865688.86 total 20.02 last 152.90) total: 4500 /unlink in 28.11 seconds: 160.11 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95447 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: modules unloaded.