== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:29:08 (1781587748) Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg134-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server oleg134-server: oleg134-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg134-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg134-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 767 (time 1781587956.65 total 10.01 last 76.62) - open/close 1471 (time 1781587966.66 total 20.02 last 70.33) - open/close 2212 (time 1781587976.66 total 30.03 last 74.06) - open/close 2838 (time 1781587986.68 total 40.04 last 62.51) - open/close 3542 (time 1781587996.68 total 50.04 last 70.38) - open/close 4247 (time 1781588006.69 total 60.06 last 70.40) total: 4500 open/close in 63.96 seconds: 70.36 ops/second - /unlink 1302 (time 1781588025.52 total 10.01 last 130.09) - /unlink 2763 (time 1781588035.52 total 20.01 last 146.07) - /unlink 4185 (time 1781588045.54 total 30.02 last 142.00) total: 4500 /unlink in 32.67 seconds: 137.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 700 (time 1781588063.60 total 10.01 last 69.95) - open/close 1589 (time 1781588073.60 total 20.01 last 88.85) - open/close 2314 (time 1781588083.61 total 30.02 last 72.43) - open/close 3091 (time 1781588093.61 total 40.02 last 77.70) - open/close 3809 (time 1781588103.62 total 50.03 last 71.73) total: 4500 open/close in 58.86 seconds: 76.45 ops/second - /unlink 1486 (time 1781588126.80 total 10.00 last 148.56) - /unlink 2946 (time 1781588136.80 total 20.01 last 145.91) - /unlink 4398 (time 1781588146.81 total 30.01 last 145.15) total: 4500 /unlink in 30.60 seconds: 147.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 742 (time 1781588162.99 total 10.01 last 74.15) - open/close 1445 (time 1781588173.00 total 20.02 last 70.23) - open/close 2185 (time 1781588183.01 total 30.02 last 73.96) - open/close 2991 (time 1781588193.02 total 40.03 last 80.52) - open/close 3725 (time 1781588203.03 total 50.04 last 73.35) - open/close 4450 (time 1781588213.03 total 60.05 last 72.46) total: 4500 open/close in 60.60 seconds: 74.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1341 (time 1781588228.90 total 10.02 last 133.89) - /unlink 2668 (time 1781588238.90 total 20.02 last 132.64) - /unlink 4111 (time 1781588248.91 total 30.02 last 144.25) total: 4500 /unlink in 33.27 seconds: 135.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 718 (time 1781588267.36 total 10.00 last 71.77) - open/close 1411 (time 1781588277.36 total 20.01 last 69.28) - open/close 2132 (time 1781588287.40 total 30.05 last 71.80) - open/close 2714 (time 1781588297.43 total 40.08 last 58.02) - open/close 3417 (time 1781588307.44 total 50.09 last 70.24) - open/close 4217 (time 1781588317.45 total 60.09 last 79.98) total: 4500 open/close in 64.00 seconds: 70.31 ops/second - /unlink 1250 (time 1781588336.58 total 10.00 last 124.98) - /unlink 2505 (time 1781588346.58 total 20.00 last 125.48) - /unlink 3838 (time 1781588356.58 total 30.01 last 133.27) total: 4500 /unlink in 35.66 seconds: 126.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 609 (time 1781588376.83 total 10.02 last 60.80) - open/close 1267 (time 1781588386.85 total 20.03 last 65.68) - open/close 1984 (time 1781588396.86 total 30.05 last 71.62) - open/close 2624 (time 1781588406.86 total 40.05 last 63.98) - open/close 3272 (time 1781588416.87 total 50.05 last 64.78) - open/close 3843 (time 1781588426.87 total 60.06 last 57.06) total: 4500 open/close in 69.17 seconds: 65.06 ops/second - /unlink 1191 (time 1781588451.44 total 10.01 last 119.02) - /unlink 2399 (time 1781588461.44 total 20.01 last 120.78) - /unlink 3704 (time 1781588471.44 total 30.01 last 130.49) total: 4500 /unlink in 36.15 seconds: 124.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 696 (time 1781588492.92 total 10.01 last 69.51) - open/close 1388 (time 1781588502.92 total 20.01 last 69.19) - open/close 2201 (time 1781588512.93 total 30.02 last 81.27) - open/close 3014 (time 1781588522.93 total 40.02 last 81.29) - open/close 3712 (time 1781588532.93 total 50.02 last 69.79) total: 4500 open/close in 59.34 seconds: 75.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1357 (time 1781588557.11 total 10.00 last 135.66) - /unlink 2760 (time 1781588567.11 total 20.01 last 140.21) - /unlink 4047 (time 1781588577.12 total 30.01 last 128.65) total: 4500 /unlink in 33.26 seconds: 135.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 712 (time 1781588595.09 total 10.01 last 71.15) - open/close 1471 (time 1781588605.10 total 20.01 last 75.87) - open/close 2231 (time 1781588615.11 total 30.02 last 75.90) - open/close 2971 (time 1781588625.11 total 40.03 last 74.00) - open/close 3669 (time 1781588635.13 total 50.04 last 69.68) - open/close 4298 (time 1781588645.13 total 60.04 last 62.89) total: 4500 open/close in 63.61 seconds: 70.75 ops/second - /unlink 1457 (time 1781588663.99 total 10.02 last 145.44) - /unlink 2830 (time 1781588673.99 total 20.02 last 137.29) - /unlink 4192 (time 1781588683.99 total 30.02 last 136.17) total: 4500 /unlink in 32.38 seconds: 138.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 676 (time 1781588701.98 total 10.01 last 67.52) - open/close 1301 (time 1781588712.00 total 20.03 last 62.39) - open/close 1953 (time 1781588722.01 total 30.04 last 65.15) - open/close 2569 (time 1781588732.02 total 40.05 last 61.54) - open/close 3154 (time 1781588742.03 total 50.06 last 58.41) - open/close 3835 (time 1781588752.05 total 60.08 last 67.99) total: 4500 open/close in 69.77 seconds: 64.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1422 (time 1781588777.14 total 10.01 last 142.00) - /unlink 2873 (time 1781588787.17 total 20.04 last 144.73) - /unlink 4017 (time 1781588797.17 total 30.04 last 114.36) total: 4500 /unlink in 33.88 seconds: 132.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 693 (time 1781588816.76 total 10.01 last 69.25) - open/close 1329 (time 1781588826.77 total 20.01 last 63.56) - open/close 1995 (time 1781588836.79 total 30.03 last 66.48) - open/close 2618 (time 1781588846.80 total 40.05 last 62.21) - open/close 3181 (time 1781588856.81 total 50.05 last 56.26) - open/close 3662 (time 1781588866.81 total 60.05 last 48.10) - open/close 4189 (time 1781588876.82 total 70.06 last 52.66) total: 4500 open/close in 76.03 seconds: 59.19 ops/second - /unlink 1075 (time 1781588900.26 total 10.01 last 107.43) - /unlink 2396 (time 1781588910.26 total 20.01 last 132.08) - /unlink 3691 (time 1781588920.27 total 30.02 last 129.34) total: 4500 /unlink in 38.32 seconds: 117.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 725 (time 1781588944.08 total 10.04 last 72.18) - open/close 1347 (time 1781588954.09 total 20.06 last 62.12) - open/close 2028 (time 1781588964.09 total 30.06 last 68.09) - open/close 2642 (time 1781588974.10 total 40.07 last 61.35) - open/close 3365 (time 1781588984.11 total 50.08 last 72.24) - open/close 3976 (time 1781588994.12 total 60.09 last 61.02) total: 4500 open/close in 67.33 seconds: 66.84 ops/second - /unlink 1214 (time 1781589017.23 total 10.04 last 120.88) - /unlink 2171 (time 1781589027.23 total 20.04 last 95.69) - /unlink 3275 (time 1781589037.23 total 30.05 last 110.39) - /unlink 4169 (time 1781589047.24 total 40.05 last 89.35) total: 4500 /unlink in 43.38 seconds: 103.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 638 (time 1781589067.62 total 10.01 last 63.76) - open/close 1365 (time 1781589077.62 total 20.01 last 72.68) - open/close 2030 (time 1781589087.64 total 30.02 last 66.41) - open/close 2857 (time 1781589097.64 total 40.02 last 82.70) - open/close 3568 (time 1781589107.65 total 50.04 last 71.02) - open/close 4177 (time 1781589117.67 total 60.05 last 60.79) total: 4500 open/close in 64.50 seconds: 69.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1152 (time 1781589139.61 total 10.00 last 115.15) - /unlink 2320 (time 1781589149.61 total 20.01 last 116.79) - /unlink 3561 (time 1781589159.62 total 30.01 last 124.06) total: 4500 /unlink in 37.01 seconds: 121.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 677 (time 1781589182.47 total 10.01 last 67.66) - open/close 1543 (time 1781589192.47 total 20.01 last 86.57) - open/close 2283 (time 1781589202.48 total 30.01 last 73.98) - open/close 2918 (time 1781589212.49 total 40.02 last 63.43) - open/close 3440 (time 1781589222.49 total 50.02 last 52.19) - open/close 4030 (time 1781589232.50 total 60.03 last 58.94) total: 4500 open/close in 67.43 seconds: 66.74 ops/second - /unlink 1268 (time 1781589255.40 total 10.00 last 126.80) - /unlink 2368 (time 1781589265.41 total 20.01 last 109.92) - /unlink 3688 (time 1781589275.42 total 30.02 last 131.88) total: 4500 /unlink in 36.71 seconds: 122.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 645 (time 1781589297.82 total 10.02 last 64.34) - open/close 1186 (time 1781589307.83 total 20.04 last 54.04) - open/close 1774 (time 1781589317.85 total 30.05 last 58.69) - open/close 2455 (time 1781589327.86 total 40.07 last 68.01) - open/close 3169 (time 1781589337.89 total 50.09 last 71.24) - open/close 3854 (time 1781589347.90 total 60.10 last 68.42) total: 4500 open/close in 68.74 seconds: 65.47 ops/second - /unlink 1267 (time 1781589371.27 total 10.00 last 126.70) - /unlink 2714 (time 1781589381.29 total 20.02 last 144.47) - /unlink 4162 (time 1781589391.29 total 30.02 last 144.75) total: 4500 /unlink in 32.77 seconds: 137.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 649 (time 1781589410.09 total 10.00 last 64.89) - open/close 1314 (time 1781589420.10 total 20.01 last 66.42) - open/close 2005 (time 1781589430.12 total 30.03 last 68.98) - open/close 2619 (time 1781589440.13 total 40.04 last 61.33) - open/close 3307 (time 1781589450.14 total 50.05 last 68.76) - open/close 3977 (time 1781589460.18 total 60.09 last 66.71) total: 4500 open/close in 68.61 seconds: 65.58 ops/second - /unlink 1179 (time 1781589485.48 total 10.01 last 117.76) - /unlink 2157 (time 1781589495.48 total 20.02 last 97.77) - /unlink 3227 (time 1781589505.49 total 30.02 last 106.91) - /unlink 4299 (time 1781589515.49 total 40.03 last 107.16) total: 4500 /unlink in 41.99 seconds: 107.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 641 (time 1781589533.53 total 10.00 last 64.09) - open/close 1327 (time 1781589543.54 total 20.01 last 68.57) - open/close 1958 (time 1781589553.55 total 30.02 last 63.04) - open/close 2510 (time 1781589563.56 total 40.02 last 55.17) - open/close 3154 (time 1781589573.57 total 50.03 last 64.32) - open/close 3716 (time 1781589583.57 total 60.04 last 56.16) - open/close 4343 (time 1781589593.59 total 70.05 last 62.61) total: 4500 open/close in 72.49 seconds: 62.08 ops/second - /unlink 1146 (time 1781589611.50 total 10.02 last 114.41) - /unlink 2432 (time 1781589621.50 total 20.02 last 128.60) - /unlink 3488 (time 1781589631.50 total 30.02 last 105.56) total: 4500 /unlink in 39.47 seconds: 114.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 645 (time 1781589657.29 total 10.03 last 64.31) - open/close 1170 (time 1781589667.30 total 20.03 last 52.48) - open/close 1837 (time 1781589677.32 total 30.06 last 66.55) - open/close 2501 (time 1781589687.34 total 40.08 last 66.25) - open/close 3160 (time 1781589697.35 total 50.09 last 65.86) - open/close 3747 (time 1781589707.35 total 60.09 last 58.69) - open/close 4377 (time 1781589717.36 total 70.09 last 62.97) total: 4500 open/close in 71.77 seconds: 62.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1323 (time 1781589734.58 total 10.00 last 132.25) - /unlink 2614 (time 1781589744.58 total 20.00 last 129.08) - /unlink 3730 (time 1781589754.59 total 30.01 last 111.55) total: 4500 /unlink in 36.19 seconds: 124.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 666 (time 1781589776.83 total 10.00 last 66.57) - open/close 1344 (time 1781589786.84 total 20.01 last 67.76) - open/close 2138 (time 1781589796.85 total 30.02 last 79.33) - open/close 2833 (time 1781589806.85 total 40.03 last 69.45) - open/close 3421 (time 1781589816.85 total 50.03 last 58.80) - open/close 3984 (time 1781589826.86 total 60.03 last 56.27) total: 4500 open/close in 68.99 seconds: 65.23 ops/second - /unlink 1253 (time 1781589850.64 total 10.00 last 125.25) - /unlink 2332 (time 1781589860.64 total 20.01 last 107.87) - /unlink 3557 (time 1781589870.65 total 30.01 last 122.43) total: 4500 /unlink in 38.39 seconds: 117.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 697 (time 1781589895.05 total 10.01 last 69.62) - open/close 1246 (time 1781589905.06 total 20.02 last 54.83) - open/close 2019 (time 1781589915.07 total 30.03 last 77.23) - open/close 2704 (time 1781589925.08 total 40.04 last 68.47) - open/close 3268 (time 1781589935.09 total 50.04 last 56.36) - open/close 3903 (time 1781589945.09 total 60.05 last 63.46) - open/close 4489 (time 1781589955.10 total 70.06 last 58.58) total: 4500 open/close in 70.17 seconds: 64.13 ops/second - /unlink 1272 (time 1781589970.35 total 10.00 last 127.14) - /unlink 2428 (time 1781589980.35 total 20.01 last 115.59) - /unlink 3665 (time 1781589990.35 total 30.01 last 123.67) total: 4500 /unlink in 37.27 seconds: 120.73 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11225: 95000 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: modules unloaded.