== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:59:19 (1788602359) Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg134-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server oleg134-server: oleg134-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg134-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg134-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 792 (time 1788602587.71 total 10.01 last 79.15) - open/close 1598 (time 1788602597.71 total 20.01 last 80.56) - open/close 2364 (time 1788602607.72 total 30.02 last 76.52) - open/close 2934 (time 1788602617.73 total 40.03 last 56.98) - open/close 3470 (time 1788602627.77 total 50.07 last 53.35) - open/close 4091 (time 1788602637.78 total 60.08 last 62.07) total: 4500 open/close in 66.94 seconds: 67.22 ops/second - /unlink 1356 (time 1788602659.31 total 10.00 last 135.54) - /unlink 2521 (time 1788602669.31 total 20.01 last 116.46) - /unlink 3733 (time 1788602679.32 total 30.01 last 121.17) total: 4500 /unlink in 35.68 seconds: 126.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 610 (time 1788602699.68 total 10.01 last 60.94) - open/close 1252 (time 1788602709.70 total 20.03 last 64.06) - open/close 1900 (time 1788602719.70 total 30.03 last 64.78) - open/close 2538 (time 1788602729.72 total 40.05 last 63.72) - open/close 3294 (time 1788602739.72 total 50.05 last 75.58) - open/close 4045 (time 1788602749.72 total 60.06 last 75.05) total: 4500 open/close in 66.34 seconds: 67.83 ops/second - /unlink 1253 (time 1788602770.74 total 10.01 last 125.12) - /unlink 2387 (time 1788602780.74 total 20.01 last 113.39) - /unlink 3648 (time 1788602790.76 total 30.03 last 125.89) total: 4500 /unlink in 38.21 seconds: 117.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 707 (time 1788602815.21 total 10.00 last 70.68) - open/close 1277 (time 1788602825.22 total 20.01 last 56.98) - open/close 2007 (time 1788602835.23 total 30.02 last 72.94) - open/close 2720 (time 1788602845.23 total 40.02 last 71.30) - open/close 3412 (time 1788602855.23 total 50.02 last 69.18) - open/close 4173 (time 1788602865.23 total 60.02 last 76.09) total: 4500 open/close in 64.65 seconds: 69.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1406 (time 1788602885.23 total 10.00 last 140.57) - /unlink 2476 (time 1788602895.23 total 20.00 last 106.97) - /unlink 3592 (time 1788602905.24 total 30.02 last 111.48) total: 4500 /unlink in 36.80 seconds: 122.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 734 (time 1788602926.70 total 10.00 last 73.39) - open/close 1426 (time 1788602936.70 total 20.00 last 69.20) - open/close 2127 (time 1788602946.71 total 30.01 last 70.05) - open/close 2727 (time 1788602956.72 total 40.02 last 59.93) - open/close 3359 (time 1788602966.73 total 50.02 last 63.18) - open/close 3929 (time 1788602976.73 total 60.03 last 56.99) total: 4500 open/close in 67.90 seconds: 66.27 ops/second - /unlink 1248 (time 1788602999.48 total 10.01 last 124.67) - /unlink 2354 (time 1788603009.49 total 20.01 last 110.58) - /unlink 3576 (time 1788603019.49 total 30.01 last 122.17) total: 4500 /unlink in 37.94 seconds: 118.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 641 (time 1788603042.96 total 10.01 last 64.02) - open/close 1374 (time 1788603052.97 total 20.03 last 73.19) - open/close 2166 (time 1788603062.98 total 30.04 last 79.11) - open/close 2915 (time 1788603072.99 total 40.04 last 74.86) - open/close 3538 (time 1788603083.01 total 50.06 last 62.20) - open/close 4240 (time 1788603093.02 total 60.07 last 70.12) total: 4500 open/close in 63.86 seconds: 70.47 ops/second - /unlink 1390 (time 1788603110.95 total 10.00 last 138.94) - /unlink 2477 (time 1788603120.97 total 20.02 last 108.51) - /unlink 3575 (time 1788603130.99 total 30.05 last 109.53) - /unlink 4423 (time 1788603141.01 total 40.07 last 84.64) total: 4500 /unlink in 40.82 seconds: 110.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 727 (time 1788603156.35 total 10.00 last 72.67) - open/close 1545 (time 1788603166.37 total 20.02 last 81.65) - open/close 2332 (time 1788603176.37 total 30.02 last 78.68) - open/close 2985 (time 1788603186.38 total 40.03 last 65.24) - open/close 3765 (time 1788603196.39 total 50.04 last 77.94) total: 4500 open/close in 60.00 seconds: 75.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1349 (time 1788603221.18 total 10.00 last 134.89) - /unlink 2726 (time 1788603231.18 total 20.00 last 137.69) - /unlink 4066 (time 1788603241.19 total 30.00 last 133.97) total: 4500 /unlink in 33.56 seconds: 134.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 694 (time 1788603260.69 total 10.01 last 69.35) - open/close 1432 (time 1788603270.70 total 20.01 last 73.78) - open/close 2186 (time 1788603280.70 total 30.01 last 75.37) - open/close 2952 (time 1788603290.71 total 40.02 last 76.55) - open/close 3637 (time 1788603300.71 total 50.03 last 68.47) - open/close 4439 (time 1788603310.74 total 60.06 last 79.96) total: 4500 open/close in 60.89 seconds: 73.91 ops/second - /unlink 1362 (time 1788603326.18 total 10.00 last 136.20) - /unlink 2691 (time 1788603336.18 total 20.00 last 132.84) - /unlink 4022 (time 1788603346.19 total 30.01 last 132.97) total: 4500 /unlink in 33.63 seconds: 133.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 628 (time 1788603365.61 total 10.00 last 62.77) - open/close 1375 (time 1788603375.62 total 20.01 last 74.66) - open/close 2181 (time 1788603385.65 total 30.04 last 80.35) - open/close 2907 (time 1788603395.65 total 40.05 last 72.56) - open/close 3760 (time 1788603405.66 total 50.05 last 85.24) total: 4500 open/close in 58.86 seconds: 76.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1266 (time 1788603428.66 total 10.00 last 126.55) - /unlink 2717 (time 1788603438.66 total 20.00 last 145.09) - /unlink 3949 (time 1788603448.66 total 30.01 last 123.14) total: 4500 /unlink in 34.10 seconds: 131.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 661 (time 1788603467.96 total 10.01 last 66.04) - open/close 1431 (time 1788603477.97 total 20.01 last 76.97) - open/close 2198 (time 1788603487.97 total 30.02 last 76.64) - open/close 2874 (time 1788603497.98 total 40.02 last 67.57) - open/close 3582 (time 1788603507.98 total 50.02 last 70.80) - open/close 4464 (time 1788603517.98 total 60.03 last 88.17) total: 4500 open/close in 60.55 seconds: 74.32 ops/second - /unlink 1443 (time 1788603533.20 total 10.00 last 144.30) - /unlink 2861 (time 1788603543.21 total 20.01 last 141.66) - /unlink 4167 (time 1788603553.21 total 30.01 last 130.56) total: 4500 /unlink in 32.67 seconds: 137.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 680 (time 1788603572.05 total 10.02 last 67.90) - open/close 1426 (time 1788603582.06 total 20.03 last 74.49) - open/close 2187 (time 1788603592.06 total 30.03 last 76.09) - open/close 2929 (time 1788603602.08 total 40.05 last 74.07) - open/close 3726 (time 1788603612.08 total 50.05 last 79.67) - open/close 4469 (time 1788603622.09 total 60.06 last 74.28) total: 4500 open/close in 60.78 seconds: 74.04 ops/second - /unlink 1391 (time 1788603638.21 total 10.01 last 138.92) - /unlink 2788 (time 1788603648.22 total 20.02 last 139.61) - /unlink 4029 (time 1788603658.23 total 30.03 last 124.00) total: 4500 /unlink in 33.54 seconds: 134.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 575 (time 1788603677.94 total 10.03 last 57.34) - open/close 1146 (time 1788603687.95 total 20.04 last 57.04) - open/close 1932 (time 1788603697.96 total 30.04 last 78.56) - open/close 2684 (time 1788603707.96 total 40.04 last 75.20) - open/close 3429 (time 1788603717.96 total 50.04 last 74.49) - open/close 4225 (time 1788603727.98 total 60.07 last 79.43) total: 4500 open/close in 63.69 seconds: 70.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1440 (time 1788603745.64 total 10.01 last 143.92) - /unlink 2846 (time 1788603755.64 total 20.01 last 140.53) - /unlink 4103 (time 1788603765.65 total 30.02 last 125.58) total: 4500 /unlink in 33.39 seconds: 134.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 649 (time 1788603784.11 total 10.01 last 64.81) - open/close 1417 (time 1788603794.12 total 20.02 last 76.74) - open/close 2158 (time 1788603804.13 total 30.03 last 74.06) - open/close 2915 (time 1788603814.13 total 40.03 last 75.65) - open/close 3552 (time 1788603824.14 total 50.04 last 63.64) - open/close 4273 (time 1788603834.16 total 60.06 last 72.00) total: 4500 open/close in 63.21 seconds: 71.20 ops/second - /unlink 1129 (time 1788603852.99 total 10.00 last 112.89) - /unlink 2080 (time 1788603863.00 total 20.00 last 95.07) - /unlink 3394 (time 1788603873.00 total 30.00 last 131.40) total: 4500 /unlink in 38.46 seconds: 116.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 688 (time 1788603896.87 total 10.03 last 68.58) - open/close 1402 (time 1788603906.87 total 20.03 last 71.39) - open/close 2190 (time 1788603916.87 total 30.04 last 78.76) - open/close 2886 (time 1788603926.88 total 40.05 last 69.55) - open/close 3664 (time 1788603936.88 total 50.05 last 77.78) - open/close 4410 (time 1788603946.89 total 60.06 last 74.50) total: 4500 open/close in 61.05 seconds: 73.71 ops/second - /unlink 1275 (time 1788603961.96 total 10.00 last 127.49) - /unlink 2327 (time 1788603971.96 total 20.00 last 105.19) - /unlink 3380 (time 1788603981.97 total 30.01 last 105.23) total: 4500 /unlink in 39.92 seconds: 112.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 734 (time 1788604006.77 total 10.01 last 73.36) - open/close 1474 (time 1788604016.78 total 20.02 last 73.93) - open/close 2245 (time 1788604026.78 total 30.02 last 77.07) - open/close 2962 (time 1788604036.79 total 40.03 last 71.64) - open/close 3587 (time 1788604046.81 total 50.04 last 62.41) - open/close 4193 (time 1788604056.81 total 60.05 last 60.56) total: 4500 open/close in 64.92 seconds: 69.32 ops/second - /unlink 1246 (time 1788604076.66 total 10.00 last 124.55) - /unlink 2502 (time 1788604086.67 total 20.01 last 125.54) - /unlink 4034 (time 1788604096.67 total 30.01 last 153.18) total: 4500 /unlink in 33.52 seconds: 134.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 665 (time 1788604115.17 total 10.02 last 66.34) - open/close 1290 (time 1788604125.19 total 20.04 last 62.41) - open/close 2000 (time 1788604135.19 total 30.04 last 70.98) - open/close 2653 (time 1788604145.19 total 40.04 last 65.29) - open/close 3405 (time 1788604155.20 total 50.05 last 75.16) - open/close 4110 (time 1788604165.21 total 60.06 last 70.40) total: 4500 open/close in 65.17 seconds: 69.05 ops/second - /unlink 1151 (time 1788604185.72 total 10.00 last 115.07) - /unlink 2483 (time 1788604195.72 total 20.00 last 133.19) - /unlink 3960 (time 1788604205.72 total 30.00 last 147.69) total: 4500 /unlink in 34.53 seconds: 130.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 623 (time 1788604226.06 total 10.01 last 62.24) - open/close 1147 (time 1788604236.07 total 20.02 last 52.36) - open/close 1888 (time 1788604246.07 total 30.02 last 74.07) - open/close 2493 (time 1788604256.11 total 40.06 last 60.25) - open/close 3177 (time 1788604266.12 total 50.07 last 68.34) - open/close 3779 (time 1788604276.13 total 60.08 last 60.12) total: 4500 open/close in 69.69 seconds: 64.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1210 (time 1788604300.22 total 10.02 last 120.77) - /unlink 2363 (time 1788604310.23 total 20.02 last 115.27) - /unlink 3537 (time 1788604320.23 total 30.03 last 117.35) total: 4500 /unlink in 37.87 seconds: 118.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 705 (time 1788604345.00 total 10.01 last 70.41) - open/close 1443 (time 1788604355.00 total 20.01 last 73.79) - open/close 2155 (time 1788604365.01 total 30.02 last 71.14) - open/close 2809 (time 1788604375.02 total 40.04 last 65.30) - open/close 3506 (time 1788604385.03 total 50.05 last 69.65) - open/close 4028 (time 1788604395.05 total 60.07 last 52.09) total: 4500 open/close in 69.96 seconds: 64.32 ops/second - /unlink 1193 (time 1788604421.61 total 10.01 last 119.24) - /unlink 2360 (time 1788604431.62 total 20.01 last 116.61) - /unlink 3697 (time 1788604441.62 total 30.02 last 133.59) total: 4500 /unlink in 36.38 seconds: 123.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 730 (time 1788604462.87 total 10.01 last 72.94) - open/close 1526 (time 1788604472.88 total 20.02 last 79.49) - open/close 2323 (time 1788604482.90 total 30.04 last 79.52) - open/close 3018 (time 1788604492.93 total 40.07 last 69.30) - open/close 3713 (time 1788604502.96 total 50.10 last 69.30) - open/close 4355 (time 1788604512.97 total 60.11 last 64.16) total: 4500 open/close in 62.54 seconds: 71.96 ops/second - /unlink 1322 (time 1788604530.92 total 10.00 last 132.17) - /unlink 2660 (time 1788604540.92 total 20.00 last 133.77) - /unlink 4054 (time 1788604550.92 total 30.01 last 139.39) total: 4500 /unlink in 33.19 seconds: 135.58 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11184: 94134 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: modules unloaded.