== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:48:31 (1787867311) Stopping clients: oleg616-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg616-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg616-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg616-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg616-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg616-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg616-server oleg616-server: oleg616-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg616-server' oleg616-server: oleg616-server.virtnet: executing load_modules_local oleg616-server: Loading modules from /home/green/git/lustre-release/lustre oleg616-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg616-server: detected 4 online CPUs by sysfs oleg616-server: MODOPTS_LIBCFS= oleg616-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg616-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg616-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg616-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 579 (time 1787867512.06 total 10.01 last 57.87) - open/close 1182 (time 1787867522.09 total 20.04 last 60.12) - open/close 1907 (time 1787867532.10 total 30.04 last 72.47) - open/close 2547 (time 1787867542.13 total 40.07 last 63.81) - open/close 3191 (time 1787867552.13 total 50.08 last 64.35) - open/close 3863 (time 1787867562.14 total 60.08 last 67.15) total: 4500 open/close in 69.83 seconds: 64.44 ops/second - /unlink 1245 (time 1787867586.77 total 10.00 last 124.45) - /unlink 2496 (time 1787867596.77 total 20.01 last 125.08) - /unlink 3692 (time 1787867606.78 total 30.01 last 119.54) total: 4500 /unlink in 36.62 seconds: 122.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 622 (time 1787867628.84 total 10.01 last 62.14) - open/close 1234 (time 1787867638.86 total 20.03 last 61.09) - open/close 1868 (time 1787867648.87 total 30.03 last 63.38) - open/close 2497 (time 1787867658.88 total 40.04 last 62.82) - open/close 3054 (time 1787867668.89 total 50.06 last 55.62) - open/close 3782 (time 1787867678.90 total 60.06 last 72.77) total: 4500 open/close in 70.00 seconds: 64.29 ops/second - /unlink 1411 (time 1787867703.97 total 10.00 last 141.05) - /unlink 2635 (time 1787867713.98 total 20.01 last 122.37) - /unlink 3966 (time 1787867723.98 total 30.01 last 133.04) total: 4500 /unlink in 34.38 seconds: 130.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 580 (time 1787867743.95 total 10.03 last 57.83) - open/close 1033 (time 1787867753.99 total 20.06 last 45.14) - open/close 1581 (time 1787867764.00 total 30.07 last 54.76) - open/close 2238 (time 1787867774.00 total 40.07 last 65.68) - open/close 2847 (time 1787867784.03 total 50.10 last 60.73) - open/close 3524 (time 1787867794.03 total 60.11 last 67.68) - open/close 4236 (time 1787867804.04 total 70.12 last 71.11) total: 4500 open/close in 74.04 seconds: 60.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1240 (time 1787867822.80 total 10.00 last 124.00) - /unlink 2529 (time 1787867832.81 total 20.00 last 128.86) - /unlink 3709 (time 1787867842.81 total 30.00 last 118.00) total: 4500 /unlink in 35.85 seconds: 125.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 734 (time 1787867863.55 total 10.01 last 73.35) - open/close 1343 (time 1787867873.56 total 20.01 last 60.87) - open/close 1940 (time 1787867883.56 total 30.02 last 59.65) - open/close 2400 (time 1787867893.58 total 40.03 last 45.94) - open/close 2936 (time 1787867903.59 total 50.04 last 53.55) - open/close 3533 (time 1787867913.59 total 60.05 last 59.67) - open/close 4194 (time 1787867923.60 total 70.06 last 66.02) total: 4500 open/close in 74.19 seconds: 60.65 ops/second - /unlink 1411 (time 1787867942.59 total 10.01 last 140.99) - /unlink 2752 (time 1787867952.59 total 20.01 last 134.08) - /unlink 4176 (time 1787867962.59 total 30.01 last 142.36) total: 4500 /unlink in 32.38 seconds: 138.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 751 (time 1787867979.67 total 10.00 last 75.07) - open/close 1463 (time 1787867989.68 total 20.01 last 71.16) - open/close 2249 (time 1787867999.70 total 30.04 last 78.38) - open/close 2879 (time 1787868009.71 total 40.05 last 62.94) - open/close 3571 (time 1787868019.73 total 50.06 last 69.10) - open/close 4286 (time 1787868029.73 total 60.06 last 71.49) total: 4500 open/close in 63.16 seconds: 71.24 ops/second - /unlink 1276 (time 1787868048.02 total 10.01 last 127.49) - /unlink 2391 (time 1787868058.02 total 20.01 last 111.48) - /unlink 3588 (time 1787868068.03 total 30.02 last 119.61) total: 4500 /unlink in 37.13 seconds: 121.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 657 (time 1787868090.52 total 10.00 last 65.69) - open/close 1348 (time 1787868100.52 total 20.01 last 69.07) - open/close 1932 (time 1787868110.53 total 30.02 last 58.35) - open/close 2566 (time 1787868120.55 total 40.03 last 63.31) - open/close 3224 (time 1787868130.55 total 50.03 last 65.79) - open/close 3754 (time 1787868140.56 total 60.04 last 52.94) - open/close 4327 (time 1787868150.56 total 70.05 last 57.26) total: 4500 open/close in 72.52 seconds: 62.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1307 (time 1787868168.03 total 10.00 last 130.69) - /unlink 2581 (time 1787868178.03 total 20.01 last 127.33) - /unlink 3968 (time 1787868188.04 total 30.01 last 138.65) total: 4500 /unlink in 33.95 seconds: 132.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 765 (time 1787868206.85 total 10.01 last 76.39) - open/close 1472 (time 1787868216.87 total 20.03 last 70.60) - open/close 2274 (time 1787868226.88 total 30.04 last 80.14) - open/close 2990 (time 1787868236.88 total 40.04 last 71.55) - open/close 3649 (time 1787868246.89 total 50.05 last 65.85) - open/close 4290 (time 1787868256.91 total 60.07 last 63.96) total: 4500 open/close in 63.79 seconds: 70.54 ops/second - /unlink 1138 (time 1787868275.35 total 10.00 last 113.76) - /unlink 1968 (time 1787868285.35 total 20.00 last 82.99) - /unlink 2972 (time 1787868295.35 total 30.01 last 100.34) - /unlink 4114 (time 1787868305.36 total 40.02 last 114.09) total: 4500 /unlink in 43.38 seconds: 103.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 452 (time 1787868325.33 total 10.02 last 45.09) - open/close 1069 (time 1787868335.34 total 20.04 last 61.63) - open/close 1888 (time 1787868345.34 total 30.04 last 81.86) - open/close 2602 (time 1787868355.35 total 40.05 last 71.37) - open/close 3370 (time 1787868365.35 total 50.05 last 76.76) - open/close 4068 (time 1787868375.36 total 60.06 last 69.77) total: 4500 open/close in 66.71 seconds: 67.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1310 (time 1787868396.82 total 10.00 last 130.98) - /unlink 2427 (time 1787868406.82 total 20.00 last 111.69) - /unlink 3773 (time 1787868416.83 total 30.00 last 134.57) total: 4500 /unlink in 35.53 seconds: 126.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 654 (time 1787868437.14 total 10.00 last 65.40) - open/close 1394 (time 1787868447.14 total 20.00 last 73.97) - open/close 2121 (time 1787868457.15 total 30.01 last 72.63) - open/close 2872 (time 1787868467.16 total 40.02 last 75.06) - open/close 3630 (time 1787868477.17 total 50.03 last 75.68) - open/close 4427 (time 1787868487.18 total 60.04 last 79.64) total: 4500 open/close in 61.04 seconds: 73.73 ops/second - /unlink 1430 (time 1787868502.14 total 10.00 last 142.97) - /unlink 2740 (time 1787868512.14 total 20.00 last 130.99) - /unlink 4173 (time 1787868522.15 total 30.01 last 143.19) total: 4500 /unlink in 32.77 seconds: 137.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 712 (time 1787868539.71 total 10.00 last 71.19) - open/close 1358 (time 1787868549.71 total 20.00 last 64.60) - open/close 1971 (time 1787868559.72 total 30.01 last 61.27) - open/close 2562 (time 1787868569.73 total 40.03 last 58.99) - open/close 3221 (time 1787868579.75 total 50.04 last 65.83) - open/close 3918 (time 1787868589.76 total 60.05 last 69.61) total: 4500 open/close in 69.71 seconds: 64.55 ops/second - /unlink 1264 (time 1787868615.70 total 10.00 last 126.40) - /unlink 2266 (time 1787868625.70 total 20.01 last 100.14) - /unlink 3479 (time 1787868635.72 total 30.03 last 121.06) total: 4500 /unlink in 38.03 seconds: 118.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 770 (time 1787868658.61 total 10.01 last 76.95) - open/close 1394 (time 1787868668.62 total 20.02 last 62.34) - open/close 1896 (time 1787868678.63 total 30.03 last 50.13) - open/close 2418 (time 1787868688.64 total 40.04 last 52.17) - open/close 3120 (time 1787868698.65 total 50.04 last 70.15) - open/close 3853 (time 1787868708.65 total 60.05 last 73.28) total: 4500 open/close in 69.18 seconds: 65.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1393 (time 1787868732.45 total 10.00 last 139.27) - /unlink 2685 (time 1787868742.46 total 20.01 last 129.13) - /unlink 4024 (time 1787868752.46 total 30.01 last 133.90) total: 4500 /unlink in 33.16 seconds: 135.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 640 (time 1787868770.39 total 10.01 last 63.92) - open/close 1296 (time 1787868780.39 total 20.01 last 65.60) - open/close 1941 (time 1787868790.39 total 30.02 last 64.46) - open/close 2569 (time 1787868800.39 total 40.02 last 62.80) - open/close 3205 (time 1787868810.40 total 50.03 last 63.54) - open/close 3865 (time 1787868820.41 total 60.03 last 65.98) total: 4500 open/close in 68.32 seconds: 65.86 ops/second - /unlink 1432 (time 1787868843.01 total 10.01 last 143.03) - /unlink 2768 (time 1787868853.01 total 20.01 last 133.59) - /unlink 4015 (time 1787868863.02 total 30.02 last 124.59) total: 4500 /unlink in 33.92 seconds: 132.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 659 (time 1787868882.16 total 10.00 last 65.90) - open/close 1315 (time 1787868892.16 total 20.00 last 65.58) - open/close 1928 (time 1787868902.19 total 30.03 last 61.16) - open/close 2567 (time 1787868912.19 total 40.03 last 63.88) - open/close 3207 (time 1787868922.20 total 50.04 last 63.91) - open/close 3807 (time 1787868932.21 total 60.05 last 59.97) - open/close 4444 (time 1787868942.21 total 70.05 last 63.69) total: 4500 open/close in 70.98 seconds: 63.40 ops/second - /unlink 1406 (time 1787868958.60 total 10.01 last 140.51) - /unlink 2572 (time 1787868968.61 total 20.02 last 116.47) - /unlink 3861 (time 1787868978.62 total 30.02 last 128.84) total: 4500 /unlink in 35.83 seconds: 125.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 668 (time 1787868999.62 total 10.01 last 66.74) - open/close 1347 (time 1787869009.63 total 20.01 last 67.87) - open/close 1996 (time 1787869019.64 total 30.02 last 64.86) - open/close 2702 (time 1787869029.64 total 40.02 last 70.60) - open/close 3401 (time 1787869039.65 total 50.03 last 69.81) - open/close 4160 (time 1787869049.66 total 60.04 last 75.84) total: 4500 open/close in 64.84 seconds: 69.40 ops/second - /unlink 1416 (time 1787869068.93 total 10.02 last 141.36) - /unlink 2681 (time 1787869078.94 total 20.02 last 126.49) - /unlink 4037 (time 1787869088.94 total 30.02 last 135.56) total: 4500 /unlink in 33.57 seconds: 134.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 698 (time 1787869108.03 total 10.01 last 69.74) - open/close 1438 (time 1787869118.03 total 20.01 last 73.99) - open/close 2176 (time 1787869128.04 total 30.02 last 73.72) - open/close 2919 (time 1787869138.05 total 40.03 last 74.21) - open/close 3740 (time 1787869148.06 total 50.04 last 82.03) - open/close 4434 (time 1787869158.07 total 60.05 last 69.31) total: 4500 open/close in 60.83 seconds: 73.98 ops/second - /unlink 1329 (time 1787869173.05 total 10.01 last 132.82) - /unlink 2759 (time 1787869183.05 total 20.01 last 142.94) - /unlink 4180 (time 1787869193.07 total 30.02 last 141.93) total: 4500 /unlink in 32.49 seconds: 138.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 787 (time 1787869209.83 total 10.02 last 78.56) - open/close 1575 (time 1787869219.84 total 20.02 last 78.77) - open/close 2278 (time 1787869229.84 total 30.02 last 70.29) - open/close 2986 (time 1787869239.84 total 40.02 last 70.79) - open/close 3753 (time 1787869249.86 total 50.05 last 76.54) - open/close 4480 (time 1787869259.87 total 60.06 last 72.61) total: 4500 open/close in 60.35 seconds: 74.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1343 (time 1787869275.06 total 10.00 last 134.29) - /unlink 2642 (time 1787869285.09 total 20.02 last 129.61) - /unlink 4027 (time 1787869295.09 total 30.03 last 138.42) total: 4500 /unlink in 33.17 seconds: 135.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 704 (time 1787869312.93 total 10.01 last 70.36) - open/close 1431 (time 1787869322.93 total 20.01 last 72.66) - open/close 2135 (time 1787869332.95 total 30.03 last 70.28) - open/close 2888 (time 1787869342.95 total 40.03 last 75.27) - open/close 3587 (time 1787869352.96 total 50.04 last 69.87) - open/close 4239 (time 1787869362.96 total 60.04 last 65.19) total: 4500 open/close in 63.76 seconds: 70.58 ops/second - /unlink 1468 (time 1787869381.22 total 10.00 last 146.74) - /unlink 2900 (time 1787869391.23 total 20.01 last 143.18) - /unlink 4141 (time 1787869401.23 total 30.01 last 124.10) total: 4500 /unlink in 32.85 seconds: 137.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 689 (time 1787869418.28 total 10.01 last 68.82) - open/close 1428 (time 1787869428.29 total 20.01 last 73.87) - open/close 2083 (time 1787869438.29 total 30.02 last 65.48) - open/close 2767 (time 1787869448.30 total 40.03 last 68.35) - open/close 3480 (time 1787869458.31 total 50.04 last 71.19) - open/close 4205 (time 1787869468.32 total 60.04 last 72.47) total: 4500 open/close in 64.20 seconds: 70.10 ops/second - /unlink 1370 (time 1787869486.86 total 10.00 last 136.97) - /unlink 2799 (time 1787869496.86 total 20.01 last 142.81) - /unlink 4234 (time 1787869506.88 total 30.02 last 143.30) total: 4500 /unlink in 31.96 seconds: 140.78 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95270 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg616-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg616-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg616-server stop mds service on oleg616-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg616-server stop mds service on oleg616-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg616-server unloading modules via unload_modules_local on: 'oleg616-server' oleg616-server: oleg616-server.virtnet: executing unload_modules_local oleg616-server: modules unloaded.