== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:10:25 (1788657025) Stopping clients: oleg626-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg626-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg626-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg626-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg626-server oleg626-server: oleg626-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: kmemleak disabled oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg626-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg626-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg626-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 724 (time 1788657226.28 total 10.01 last 72.30) - open/close 1414 (time 1788657236.29 total 20.02 last 68.94) - open/close 2111 (time 1788657246.29 total 30.03 last 69.67) - open/close 2857 (time 1788657256.30 total 40.03 last 74.55) - open/close 3510 (time 1788657266.30 total 50.04 last 65.26) - open/close 4260 (time 1788657276.30 total 60.04 last 74.98) total: 4500 open/close in 63.07 seconds: 71.35 ops/second - /unlink 1147 (time 1788657293.94 total 10.00 last 114.70) - /unlink 2556 (time 1788657303.96 total 20.01 last 140.70) - /unlink 3798 (time 1788657313.96 total 30.02 last 124.16) total: 4500 /unlink in 34.93 seconds: 128.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 711 (time 1788657334.16 total 10.01 last 71.00) - open/close 1441 (time 1788657344.17 total 20.02 last 72.95) - open/close 2151 (time 1788657354.18 total 30.03 last 70.91) - open/close 2949 (time 1788657364.19 total 40.04 last 79.73) - open/close 3667 (time 1788657374.19 total 50.05 last 71.78) - open/close 4410 (time 1788657384.20 total 60.05 last 74.26) total: 4500 open/close in 61.38 seconds: 73.32 ops/second - /unlink 1329 (time 1788657399.55 total 10.00 last 132.88) - /unlink 2678 (time 1788657409.55 total 20.01 last 134.82) - /unlink 4107 (time 1788657419.56 total 30.01 last 142.81) total: 4500 /unlink in 33.25 seconds: 135.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 666 (time 1788657438.09 total 10.00 last 66.57) - open/close 1335 (time 1788657448.10 total 20.01 last 66.86) - open/close 2060 (time 1788657458.12 total 30.03 last 72.36) - open/close 2777 (time 1788657468.13 total 40.04 last 71.61) - open/close 3507 (time 1788657478.14 total 50.05 last 72.95) - open/close 4162 (time 1788657488.14 total 60.05 last 65.48) total: 4500 open/close in 64.90 seconds: 69.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1336 (time 1788657507.39 total 10.00 last 133.55) - /unlink 2657 (time 1788657517.39 total 20.00 last 132.10) - /unlink 4110 (time 1788657527.39 total 30.01 last 145.27) total: 4500 /unlink in 33.20 seconds: 135.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 754 (time 1788657545.36 total 10.01 last 75.36) - open/close 1423 (time 1788657555.36 total 20.01 last 66.87) - open/close 2155 (time 1788657565.37 total 30.02 last 73.13) - open/close 2888 (time 1788657575.39 total 40.03 last 73.19) - open/close 3668 (time 1788657585.40 total 50.05 last 77.88) - open/close 4401 (time 1788657595.41 total 60.06 last 73.27) total: 4500 open/close in 61.42 seconds: 73.26 ops/second - /unlink 1303 (time 1788657611.30 total 10.00 last 130.29) - /unlink 2632 (time 1788657621.31 total 20.00 last 132.87) - /unlink 4102 (time 1788657631.31 total 30.00 last 146.98) total: 4500 /unlink in 33.22 seconds: 135.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 781 (time 1788657649.90 total 10.00 last 78.07) - open/close 1478 (time 1788657659.91 total 20.01 last 69.64) - open/close 2153 (time 1788657669.92 total 30.02 last 67.43) - open/close 2962 (time 1788657679.94 total 40.04 last 80.78) - open/close 3698 (time 1788657689.95 total 50.05 last 73.50) - open/close 4438 (time 1788657699.95 total 60.05 last 73.98) total: 4500 open/close in 60.71 seconds: 74.13 ops/second - /unlink 1335 (time 1788657714.59 total 10.00 last 133.49) - /unlink 2675 (time 1788657724.59 total 20.01 last 133.94) - /unlink 4044 (time 1788657734.60 total 30.01 last 136.85) total: 4500 /unlink in 33.25 seconds: 135.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 742 (time 1788657752.77 total 10.00 last 74.19) - open/close 1423 (time 1788657762.78 total 20.00 last 68.09) - open/close 2148 (time 1788657772.78 total 30.00 last 72.49) - open/close 2699 (time 1788657782.85 total 40.08 last 54.69) - open/close 3415 (time 1788657792.87 total 50.09 last 71.50) - open/close 4171 (time 1788657802.87 total 60.09 last 75.60) total: 4500 open/close in 64.20 seconds: 70.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1385 (time 1788657821.14 total 10.00 last 138.46) - /unlink 2876 (time 1788657831.14 total 20.01 last 149.05) - /unlink 4292 (time 1788657841.14 total 30.01 last 141.55) total: 4500 /unlink in 31.47 seconds: 143.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 674 (time 1788657856.89 total 10.01 last 67.36) - open/close 1382 (time 1788657866.91 total 20.02 last 70.71) - open/close 2151 (time 1788657876.91 total 30.02 last 76.88) - open/close 2882 (time 1788657886.91 total 40.02 last 73.10) - open/close 3634 (time 1788657896.91 total 50.03 last 75.16) - open/close 4251 (time 1788657906.95 total 60.06 last 61.48) total: 4500 open/close in 64.92 seconds: 69.31 ops/second - /unlink 1406 (time 1788657927.15 total 10.02 last 140.37) - /unlink 2516 (time 1788657937.15 total 20.02 last 110.94) - /unlink 3816 (time 1788657947.16 total 30.03 last 129.95) total: 4500 /unlink in 35.69 seconds: 126.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 692 (time 1788657967.12 total 10.01 last 69.14) - open/close 1348 (time 1788657977.12 total 20.01 last 65.58) - open/close 2043 (time 1788657987.13 total 30.01 last 69.48) - open/close 2716 (time 1788657997.14 total 40.03 last 67.19) - open/close 3442 (time 1788658007.14 total 50.03 last 72.58) - open/close 4110 (time 1788658017.14 total 60.03 last 66.80) total: 4500 open/close in 66.20 seconds: 67.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1365 (time 1788658038.29 total 10.00 last 136.46) - /unlink 2689 (time 1788658048.30 total 20.01 last 132.36) - /unlink 3913 (time 1788658058.30 total 30.01 last 122.35) total: 4500 /unlink in 34.73 seconds: 129.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 637 (time 1788658078.24 total 10.05 last 63.40) - open/close 1155 (time 1788658088.27 total 20.07 last 51.68) - open/close 1813 (time 1788658098.28 total 30.08 last 65.74) - open/close 2455 (time 1788658108.29 total 40.09 last 64.11) - open/close 3127 (time 1788658118.30 total 50.10 last 67.15) - open/close 3850 (time 1788658128.31 total 60.11 last 72.22) total: 4500 open/close in 69.46 seconds: 64.79 ops/second - /unlink 1296 (time 1788658152.12 total 10.00 last 129.57) - /unlink 2490 (time 1788658162.12 total 20.00 last 119.37) - /unlink 3564 (time 1788658172.13 total 30.01 last 107.36) total: 4500 /unlink in 37.76 seconds: 119.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 659 (time 1788658194.91 total 10.01 last 65.82) - open/close 1248 (time 1788658204.94 total 20.04 last 58.75) - open/close 1942 (time 1788658214.95 total 30.05 last 69.34) - open/close 2631 (time 1788658224.95 total 40.05 last 68.86) - open/close 3282 (time 1788658234.97 total 50.07 last 65.00) - open/close 3960 (time 1788658244.99 total 60.09 last 67.62) total: 4500 open/close in 67.78 seconds: 66.39 ops/second - /unlink 1350 (time 1788658267.30 total 10.00 last 134.97) - /unlink 2603 (time 1788658277.31 total 20.00 last 125.27) - /unlink 3866 (time 1788658287.32 total 30.01 last 126.18) total: 4500 /unlink in 34.67 seconds: 129.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 637 (time 1788658306.08 total 10.00 last 63.69) - open/close 1319 (time 1788658316.08 total 20.00 last 68.18) - open/close 2016 (time 1788658326.08 total 30.01 last 69.67) - open/close 2695 (time 1788658336.10 total 40.02 last 67.80) - open/close 3385 (time 1788658346.12 total 50.04 last 68.89) - open/close 4083 (time 1788658356.13 total 60.05 last 69.71) total: 4500 open/close in 66.46 seconds: 67.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1322 (time 1788658377.13 total 10.00 last 132.16) - /unlink 2605 (time 1788658387.14 total 20.01 last 128.27) - /unlink 3885 (time 1788658397.14 total 30.01 last 127.94) total: 4500 /unlink in 34.74 seconds: 129.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 675 (time 1788658416.14 total 10.00 last 67.49) - open/close 1317 (time 1788658426.14 total 20.00 last 64.18) - open/close 1969 (time 1788658436.16 total 30.02 last 65.10) - open/close 2651 (time 1788658446.16 total 40.02 last 68.17) - open/close 3349 (time 1788658456.17 total 50.03 last 69.76) - open/close 4099 (time 1788658466.17 total 60.03 last 74.99) total: 4500 open/close in 65.81 seconds: 68.38 ops/second - /unlink 1244 (time 1788658485.98 total 10.00 last 124.36) - /unlink 2554 (time 1788658495.98 total 20.00 last 130.99) - /unlink 3812 (time 1788658505.98 total 30.00 last 125.79) total: 4500 /unlink in 35.59 seconds: 126.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 710 (time 1788658526.26 total 10.00 last 70.99) - open/close 1394 (time 1788658536.27 total 20.01 last 68.34) - open/close 2092 (time 1788658546.28 total 30.02 last 69.75) - open/close 2778 (time 1788658556.32 total 40.06 last 68.30) - open/close 3484 (time 1788658566.33 total 50.07 last 70.53) - open/close 4173 (time 1788658576.35 total 60.09 last 68.80) total: 4500 open/close in 64.75 seconds: 69.50 ops/second - /unlink 1482 (time 1788658594.91 total 10.01 last 148.06) - /unlink 2930 (time 1788658604.91 total 20.01 last 144.80) - /unlink 4180 (time 1788658614.92 total 30.02 last 124.90) total: 4500 /unlink in 33.14 seconds: 135.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 593 (time 1788658634.12 total 10.02 last 59.18) - open/close 1284 (time 1788658644.13 total 20.03 last 69.06) - open/close 2000 (time 1788658654.14 total 30.03 last 71.54) - open/close 2805 (time 1788658664.14 total 40.04 last 80.48) - open/close 3539 (time 1788658674.15 total 50.05 last 73.33) - open/close 4206 (time 1788658684.16 total 60.05 last 66.66) total: 4500 open/close in 64.78 seconds: 69.47 ops/second - /unlink 1222 (time 1788658703.93 total 10.00 last 122.18) - /unlink 2400 (time 1788658713.93 total 20.00 last 117.78) - /unlink 3678 (time 1788658723.94 total 30.01 last 127.72) total: 4500 /unlink in 36.36 seconds: 123.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 686 (time 1788658745.00 total 10.02 last 68.45) - open/close 1245 (time 1788658755.07 total 20.09 last 55.52) - open/close 1956 (time 1788658765.07 total 30.09 last 71.08) - open/close 2647 (time 1788658775.08 total 40.10 last 69.03) - open/close 3407 (time 1788658785.09 total 50.11 last 75.94) - open/close 4176 (time 1788658795.09 total 60.11 last 76.89) total: 4500 open/close in 64.13 seconds: 70.17 ops/second - /unlink 1485 (time 1788658813.00 total 10.00 last 148.44) - /unlink 2940 (time 1788658823.00 total 20.00 last 145.50) - /unlink 4359 (time 1788658833.00 total 30.01 last 141.83) total: 4500 /unlink in 31.02 seconds: 145.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 751 (time 1788658847.64 total 10.01 last 75.04) - open/close 1495 (time 1788658857.64 total 20.01 last 74.35) - open/close 2292 (time 1788658867.65 total 30.02 last 79.69) - open/close 3018 (time 1788658877.67 total 40.04 last 72.39) - open/close 3790 (time 1788658887.69 total 50.06 last 77.09) - open/close 4478 (time 1788658897.69 total 60.06 last 68.77) total: 4500 open/close in 60.29 seconds: 74.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1333 (time 1788658912.79 total 10.00 last 133.28) - /unlink 2595 (time 1788658922.80 total 20.01 last 126.14) - /unlink 3885 (time 1788658932.80 total 30.01 last 128.97) total: 4500 /unlink in 34.40 seconds: 130.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 717 (time 1788658952.19 total 10.01 last 71.63) - open/close 1399 (time 1788658962.19 total 20.01 last 68.19) - open/close 2014 (time 1788658972.20 total 30.01 last 61.48) - open/close 2718 (time 1788658982.20 total 40.02 last 70.38) - open/close 3393 (time 1788658992.21 total 50.02 last 67.45) - open/close 4124 (time 1788659002.21 total 60.03 last 73.07) total: 4500 open/close in 65.27 seconds: 68.95 ops/second - /unlink 1336 (time 1788659023.20 total 10.00 last 133.55) - /unlink 2727 (time 1788659033.20 total 20.00 last 139.09) - /unlink 4074 (time 1788659043.20 total 30.01 last 134.69) total: 4500 /unlink in 33.33 seconds: 135.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 653 (time 1788659062.37 total 10.01 last 65.23) - open/close 1339 (time 1788659072.37 total 20.01 last 68.58) - open/close 2029 (time 1788659082.38 total 30.02 last 68.96) - open/close 2785 (time 1788659092.38 total 40.03 last 75.55) - open/close 3466 (time 1788659102.39 total 50.03 last 68.08) - open/close 4105 (time 1788659112.40 total 60.04 last 63.82) total: 4500 open/close in 66.06 seconds: 68.12 ops/second - /unlink 1340 (time 1788659133.84 total 10.01 last 133.93) - /unlink 2584 (time 1788659143.84 total 20.01 last 124.39) - /unlink 4017 (time 1788659153.85 total 30.01 last 143.22) total: 4500 /unlink in 33.69 seconds: 133.58 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95234 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg626-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server stop mds service on oleg626-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg626-server stop mds service on oleg626-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg626-server unloading modules via unload_modules_local on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing unload_modules_local oleg626-server: kmemleak disabled modules unloaded.