== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:27:05 (1783139225) Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg425-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg425-server oleg425-server: oleg425-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg425-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg425-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 653 (time 1783139422.23 total 10.00 last 65.29) - open/close 1316 (time 1783139432.24 total 20.00 last 66.28) - open/close 1967 (time 1783139442.24 total 30.00 last 65.10) - open/close 2516 (time 1783139452.24 total 40.01 last 54.89) - open/close 3144 (time 1783139462.25 total 50.01 last 62.75) - open/close 3800 (time 1783139472.25 total 60.02 last 65.54) - open/close 4472 (time 1783139482.26 total 70.03 last 67.17) total: 4500 open/close in 70.49 seconds: 63.84 ops/second - /unlink 1283 (time 1783139497.11 total 10.01 last 128.21) - /unlink 2470 (time 1783139507.11 total 20.01 last 118.69) - /unlink 3588 (time 1783139517.11 total 30.01 last 111.79) total: 4500 /unlink in 38.09 seconds: 118.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 592 (time 1783139540.00 total 10.02 last 59.10) - open/close 1216 (time 1783139550.00 total 20.02 last 62.35) - open/close 1853 (time 1783139560.01 total 30.03 last 63.64) - open/close 2491 (time 1783139570.02 total 40.04 last 63.76) - open/close 3135 (time 1783139580.02 total 50.04 last 64.39) - open/close 3764 (time 1783139590.03 total 60.05 last 62.87) - open/close 4387 (time 1783139600.04 total 70.06 last 62.21) total: 4500 open/close in 71.63 seconds: 62.82 ops/second - /unlink 1189 (time 1783139616.07 total 10.00 last 118.88) - /unlink 2289 (time 1783139626.10 total 20.03 last 109.69) - /unlink 3395 (time 1783139636.10 total 30.03 last 110.57) total: 4500 /unlink in 39.65 seconds: 113.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 572 (time 1783139659.67 total 10.02 last 57.11) - open/close 1231 (time 1783139669.67 total 20.02 last 65.90) - open/close 1858 (time 1783139679.68 total 30.03 last 62.61) - open/close 2563 (time 1783139689.71 total 40.06 last 70.31) - open/close 3197 (time 1783139699.71 total 50.06 last 63.39) - open/close 3843 (time 1783139709.71 total 60.06 last 64.59) - open/close 4492 (time 1783139719.71 total 70.06 last 64.90) total: 4500 open/close in 70.17 seconds: 64.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1140 (time 1783139733.80 total 10.00 last 113.96) - /unlink 2396 (time 1783139743.80 total 20.01 last 125.58) - /unlink 3694 (time 1783139753.81 total 30.01 last 129.72) total: 4500 /unlink in 36.61 seconds: 122.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 656 (time 1783139774.49 total 10.04 last 65.35) - open/close 1294 (time 1783139784.51 total 20.06 last 63.66) - open/close 1975 (time 1783139794.53 total 30.08 last 67.99) - open/close 2685 (time 1783139804.54 total 40.09 last 70.91) - open/close 3266 (time 1783139814.57 total 50.11 last 57.96) - open/close 3905 (time 1783139824.57 total 60.12 last 63.89) - open/close 4491 (time 1783139834.57 total 70.12 last 58.57) total: 4500 open/close in 70.23 seconds: 64.07 ops/second - /unlink 1180 (time 1783139849.46 total 10.01 last 117.93) - /unlink 2401 (time 1783139859.47 total 20.01 last 122.03) - /unlink 3484 (time 1783139869.47 total 30.01 last 108.30) - /unlink 4493 (time 1783139879.47 total 40.02 last 100.84) total: 4500 /unlink in 40.06 seconds: 112.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 566 (time 1783139895.02 total 10.01 last 56.56) - open/close 1155 (time 1783139905.03 total 20.02 last 58.83) - open/close 1718 (time 1783139915.06 total 30.05 last 56.14) - open/close 2263 (time 1783139925.07 total 40.06 last 54.45) - open/close 2891 (time 1783139935.07 total 50.06 last 62.79) - open/close 3473 (time 1783139945.08 total 60.07 last 58.13) - open/close 4129 (time 1783139955.10 total 70.09 last 65.47) total: 4500 open/close in 76.29 seconds: 58.99 ops/second - /unlink 1245 (time 1783139975.95 total 10.01 last 124.42) - /unlink 2364 (time 1783139985.96 total 20.01 last 111.83) - /unlink 3587 (time 1783139995.97 total 30.02 last 122.23) total: 4500 /unlink in 37.54 seconds: 119.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 647 (time 1783140018.27 total 10.01 last 64.62) - open/close 1229 (time 1783140028.27 total 20.02 last 58.19) - open/close 1865 (time 1783140038.27 total 30.02 last 63.59) - open/close 2455 (time 1783140048.28 total 40.03 last 58.94) - open/close 2980 (time 1783140058.29 total 50.04 last 52.43) - open/close 3516 (time 1783140068.31 total 60.05 last 53.53) - open/close 4159 (time 1783140078.31 total 70.06 last 64.25) total: 4500 open/close in 75.25 seconds: 59.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1180 (time 1783140098.06 total 10.00 last 117.99) - /unlink 2429 (time 1783140108.07 total 20.02 last 124.71) - /unlink 3490 (time 1783140118.07 total 30.02 last 106.08) total: 4500 /unlink in 38.91 seconds: 115.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 684 (time 1783140141.97 total 10.01 last 68.36) - open/close 1275 (time 1783140151.97 total 20.01 last 59.09) - open/close 1929 (time 1783140161.97 total 30.01 last 65.38) - open/close 2587 (time 1783140171.97 total 40.01 last 65.79) - open/close 3222 (time 1783140181.99 total 50.03 last 63.40) - open/close 3885 (time 1783140191.99 total 60.04 last 66.26) total: 4500 open/close in 69.53 seconds: 64.72 ops/second - /unlink 1165 (time 1783140215.91 total 10.00 last 116.47) - /unlink 2245 (time 1783140225.91 total 20.01 last 107.95) - /unlink 3232 (time 1783140235.91 total 30.01 last 98.67) - /unlink 4196 (time 1783140245.92 total 40.02 last 96.33) total: 4500 /unlink in 42.86 seconds: 104.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 602 (time 1783140264.18 total 10.01 last 60.16) - open/close 1216 (time 1783140274.20 total 20.02 last 61.29) - open/close 1833 (time 1783140284.21 total 30.04 last 61.63) - open/close 2449 (time 1783140294.22 total 40.05 last 61.53) - open/close 3037 (time 1783140304.22 total 50.05 last 58.78) - open/close 3645 (time 1783140314.23 total 60.05 last 60.78) - open/close 4289 (time 1783140324.25 total 70.07 last 64.29) total: 4500 open/close in 73.18 seconds: 61.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1214 (time 1783140341.39 total 10.01 last 121.30) - /unlink 2462 (time 1783140351.39 total 20.01 last 124.76) - /unlink 3657 (time 1783140361.39 total 30.01 last 119.50) total: 4500 /unlink in 36.94 seconds: 121.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 684 (time 1783140382.56 total 10.01 last 68.35) - open/close 1303 (time 1783140392.56 total 20.01 last 61.87) - open/close 1963 (time 1783140402.57 total 30.02 last 65.94) - open/close 2587 (time 1783140412.57 total 40.02 last 62.39) - open/close 3268 (time 1783140422.58 total 50.03 last 68.03) - open/close 3922 (time 1783140432.59 total 60.04 last 65.33) total: 4500 open/close in 69.06 seconds: 65.16 ops/second - /unlink 1257 (time 1783140456.39 total 10.00 last 125.66) - /unlink 2428 (time 1783140466.39 total 20.01 last 117.04) - /unlink 3399 (time 1783140476.40 total 30.01 last 97.04) total: 4500 /unlink in 39.64 seconds: 113.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 636 (time 1783140500.79 total 10.01 last 63.54) - open/close 1252 (time 1783140510.80 total 20.02 last 61.57) - open/close 1945 (time 1783140520.81 total 30.03 last 69.18) - open/close 2637 (time 1783140530.82 total 40.04 last 69.18) - open/close 3326 (time 1783140540.84 total 50.06 last 68.75) - open/close 3990 (time 1783140550.85 total 60.07 last 66.31) total: 4500 open/close in 68.45 seconds: 65.74 ops/second - /unlink 1171 (time 1783140573.39 total 10.01 last 116.97) - /unlink 2382 (time 1783140583.40 total 20.02 last 121.03) - /unlink 3663 (time 1783140593.40 total 30.02 last 128.05) total: 4500 /unlink in 36.82 seconds: 122.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 707 (time 1783140614.28 total 10.01 last 70.66) - open/close 1406 (time 1783140624.29 total 20.02 last 69.82) - open/close 2065 (time 1783140634.29 total 30.02 last 65.89) - open/close 2731 (time 1783140644.31 total 40.03 last 66.52) - open/close 3373 (time 1783140654.32 total 50.05 last 64.09) - open/close 4047 (time 1783140664.33 total 60.05 last 67.36) total: 4500 open/close in 66.83 seconds: 67.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1296 (time 1783140685.38 total 10.00 last 129.60) - /unlink 2504 (time 1783140695.38 total 20.00 last 120.78) - /unlink 3763 (time 1783140705.39 total 30.01 last 125.83) total: 4500 /unlink in 36.33 seconds: 123.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 616 (time 1783140725.67 total 10.02 last 61.49) - open/close 1267 (time 1783140735.68 total 20.03 last 65.05) - open/close 1946 (time 1783140745.68 total 30.03 last 67.87) - open/close 2640 (time 1783140755.69 total 40.04 last 69.35) - open/close 3359 (time 1783140765.70 total 50.04 last 71.86) - open/close 4085 (time 1783140775.70 total 60.05 last 72.56) total: 4500 open/close in 66.49 seconds: 67.68 ops/second - /unlink 1252 (time 1783140796.42 total 10.00 last 125.14) - /unlink 2537 (time 1783140806.42 total 20.01 last 128.47) - /unlink 3824 (time 1783140816.43 total 30.01 last 128.66) total: 4500 /unlink in 34.99 seconds: 128.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 664 (time 1783140835.64 total 10.01 last 66.35) - open/close 1326 (time 1783140845.65 total 20.01 last 66.15) - open/close 1987 (time 1783140855.66 total 30.02 last 66.04) - open/close 2664 (time 1783140865.68 total 40.05 last 67.54) - open/close 3361 (time 1783140875.68 total 50.05 last 69.69) - open/close 4032 (time 1783140885.69 total 60.05 last 67.07) total: 4500 open/close in 66.68 seconds: 67.49 ops/second - /unlink 1255 (time 1783140905.80 total 10.00 last 125.48) - /unlink 2528 (time 1783140915.80 total 20.01 last 127.21) - /unlink 3861 (time 1783140925.81 total 30.01 last 133.22) total: 4500 /unlink in 35.40 seconds: 127.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 706 (time 1783140945.07 total 10.00 last 70.59) - open/close 1433 (time 1783140955.07 total 20.00 last 72.69) - open/close 2147 (time 1783140965.08 total 30.01 last 71.38) - open/close 2901 (time 1783140975.08 total 40.01 last 75.40) - open/close 3614 (time 1783140985.10 total 50.03 last 71.13) - open/close 4340 (time 1783140995.11 total 60.04 last 72.54) total: 4500 open/close in 62.60 seconds: 71.89 ops/second - /unlink 1429 (time 1783141011.07 total 10.01 last 142.77) - /unlink 2710 (time 1783141021.08 total 20.02 last 127.98) - /unlink 4014 (time 1783141031.08 total 30.02 last 130.39) total: 4500 /unlink in 33.84 seconds: 132.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 716 (time 1783141049.09 total 10.01 last 71.54) - open/close 1540 (time 1783141059.10 total 20.02 last 82.33) - open/close 2372 (time 1783141069.11 total 30.02 last 83.18) - open/close 3166 (time 1783141079.12 total 40.03 last 79.29) - open/close 3976 (time 1783141089.12 total 50.03 last 81.00) total: 4500 open/close in 56.63 seconds: 79.47 ops/second - /unlink 1469 (time 1783141108.84 total 10.00 last 146.89) - /unlink 2862 (time 1783141118.84 total 20.00 last 139.26) - /unlink 4088 (time 1783141128.86 total 30.02 last 122.45) total: 4500 /unlink in 33.60 seconds: 133.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 628 (time 1783141147.50 total 10.01 last 62.77) - open/close 1297 (time 1783141157.51 total 20.01 last 66.84) - open/close 1960 (time 1783141167.52 total 30.02 last 66.23) - open/close 2644 (time 1783141177.53 total 40.03 last 68.36) - open/close 3382 (time 1783141187.54 total 50.05 last 73.67) - open/close 4102 (time 1783141197.55 total 60.06 last 71.93) total: 4500 open/close in 65.18 seconds: 69.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1533 (time 1783141215.61 total 10.01 last 153.13) - /unlink 2876 (time 1783141225.61 total 20.01 last 134.27) - /unlink 4255 (time 1783141235.62 total 30.02 last 137.80) total: 4500 /unlink in 31.79 seconds: 141.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 807 (time 1783141250.50 total 10.00 last 80.68) - open/close 1627 (time 1783141260.51 total 20.01 last 81.96) - open/close 2401 (time 1783141270.52 total 30.02 last 77.29) - open/close 3166 (time 1783141280.53 total 40.03 last 76.42) - open/close 3963 (time 1783141290.55 total 50.04 last 79.60) total: 4500 open/close in 56.73 seconds: 79.33 ops/second - /unlink 1548 (time 1783141310.27 total 10.00 last 154.75) - /unlink 3141 (time 1783141320.27 total 20.01 last 159.26) total: 4500 /unlink in 28.86 seconds: 155.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 768 (time 1783141342.67 total 10.02 last 76.64) - open/close 1527 (time 1783141352.68 total 20.03 last 75.83) - open/close 2278 (time 1783141362.68 total 30.03 last 75.09) - open/close 3100 (time 1783141372.69 total 40.04 last 82.11) - open/close 3908 (time 1783141382.71 total 50.05 last 80.71) total: 4500 open/close in 57.75 seconds: 77.92 ops/second - /unlink 1529 (time 1783141404.08 total 10.01 last 152.75) - /unlink 2904 (time 1783141414.09 total 20.01 last 137.44) - /unlink 4311 (time 1783141424.10 total 30.02 last 140.60) total: 4500 /unlink in 31.49 seconds: 142.92 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95030 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: modules unloaded.