== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:32:43 (1782232363) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg238-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg238-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 993 (time 1782232530.50 total 10.00 last 99.26) - open/close 1852 (time 1782232540.50 total 20.01 last 85.85) - open/close 2747 (time 1782232550.50 total 30.01 last 89.49) - open/close 3636 (time 1782232560.51 total 40.01 last 88.87) total: 4500 open/close in 49.60 seconds: 90.73 ops/second - /unlink 1595 (time 1782232583.55 total 10.00 last 159.50) - /unlink 2812 (time 1782232593.56 total 20.00 last 121.67) - /unlink 3676 (time 1782232603.56 total 30.01 last 86.37) total: 4500 /unlink in 35.49 seconds: 126.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 569 (time 1782232623.45 total 10.02 last 56.81) - open/close 1011 (time 1782232633.56 total 20.12 last 43.73) - open/close 1309 (time 1782232643.60 total 30.17 last 29.67) - open/close 1852 (time 1782232653.61 total 40.17 last 54.28) - open/close 2577 (time 1782232663.61 total 50.17 last 72.49) - open/close 3469 (time 1782232673.62 total 60.18 last 89.12) - open/close 4307 (time 1782232683.63 total 70.20 last 83.67) total: 4500 open/close in 72.45 seconds: 62.11 ops/second - /unlink 1353 (time 1782232700.59 total 10.00 last 135.27) - /unlink 2747 (time 1782232710.59 total 20.00 last 139.38) - /unlink 4430 (time 1782232720.60 total 30.01 last 168.24) total: 4500 /unlink in 30.51 seconds: 147.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 869 (time 1782232735.02 total 10.01 last 86.84) - open/close 1536 (time 1782232745.03 total 20.01 last 66.66) - open/close 2487 (time 1782232755.03 total 30.01 last 95.08) - open/close 3431 (time 1782232765.03 total 40.02 last 94.37) - open/close 4323 (time 1782232775.04 total 50.03 last 89.12) total: 4500 open/close in 52.31 seconds: 86.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1528 (time 1782232791.55 total 10.00 last 152.80) - /unlink 3097 (time 1782232801.56 total 20.00 last 156.84) total: 4500 /unlink in 29.91 seconds: 150.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 779 (time 1782232825.44 total 10.01 last 77.85) - open/close 1496 (time 1782232835.44 total 20.01 last 71.70) - open/close 2273 (time 1782232845.45 total 30.02 last 77.60) - open/close 3029 (time 1782232855.47 total 40.04 last 75.45) - open/close 3794 (time 1782232865.48 total 50.05 last 76.42) total: 4500 open/close in 58.24 seconds: 77.26 ops/second - /unlink 1364 (time 1782232887.83 total 10.00 last 136.35) - /unlink 2774 (time 1782232897.84 total 20.01 last 140.92) - /unlink 4267 (time 1782232907.84 total 30.01 last 149.30) total: 4500 /unlink in 31.75 seconds: 141.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 907 (time 1782232923.88 total 10.00 last 90.66) - open/close 1750 (time 1782232933.91 total 20.03 last 84.09) - open/close 2406 (time 1782232943.91 total 30.03 last 65.57) - open/close 3208 (time 1782232953.92 total 40.05 last 80.09) - open/close 3791 (time 1782232963.93 total 50.06 last 58.25) total: 4500 open/close in 59.31 seconds: 75.88 ops/second - /unlink 1364 (time 1782232988.58 total 10.00 last 136.39) - /unlink 2581 (time 1782232998.58 total 20.00 last 121.70) - /unlink 3683 (time 1782233008.58 total 30.00 last 110.19) total: 4500 /unlink in 39.05 seconds: 115.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 802 (time 1782233034.46 total 10.03 last 79.97) - open/close 1578 (time 1782233044.48 total 20.05 last 77.45) - open/close 2313 (time 1782233054.48 total 30.06 last 73.45) - open/close 2994 (time 1782233064.49 total 40.06 last 68.07) - open/close 3707 (time 1782233074.50 total 50.07 last 71.25) - open/close 4412 (time 1782233084.52 total 60.09 last 70.35) total: 4500 open/close in 61.57 seconds: 73.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1411 (time 1782233101.34 total 10.01 last 140.99) - /unlink 2607 (time 1782233111.34 total 20.01 last 119.53) - /unlink 4034 (time 1782233121.35 total 30.02 last 142.67) total: 4500 /unlink in 33.66 seconds: 133.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 765 (time 1782233139.69 total 10.00 last 76.49) - open/close 1624 (time 1782233149.70 total 20.01 last 85.81) - open/close 2518 (time 1782233159.71 total 30.02 last 89.33) - open/close 3271 (time 1782233169.71 total 40.02 last 75.28) - open/close 3975 (time 1782233179.72 total 50.03 last 70.37) total: 4500 open/close in 58.02 seconds: 77.56 ops/second - /unlink 1278 (time 1782233202.60 total 10.00 last 127.76) - /unlink 2759 (time 1782233212.60 total 20.01 last 148.04) - /unlink 4084 (time 1782233222.61 total 30.01 last 132.45) total: 4500 /unlink in 33.05 seconds: 136.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 881 (time 1782233241.08 total 10.01 last 88.03) - open/close 1748 (time 1782233251.08 total 20.01 last 86.69) - open/close 2627 (time 1782233261.10 total 30.02 last 87.79) - open/close 3522 (time 1782233271.10 total 40.03 last 89.46) - open/close 4476 (time 1782233281.11 total 50.04 last 95.28) total: 4500 open/close in 50.27 seconds: 89.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1354 (time 1782233296.45 total 10.00 last 135.39) - /unlink 2846 (time 1782233306.47 total 20.02 last 148.91) - /unlink 4135 (time 1782233316.48 total 30.03 last 128.74) total: 4500 /unlink in 33.04 seconds: 136.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 814 (time 1782233336.13 total 10.00 last 81.38) - open/close 1691 (time 1782233346.16 total 20.03 last 87.43) - open/close 2520 (time 1782233356.17 total 30.04 last 82.87) - open/close 3440 (time 1782233366.17 total 40.04 last 91.95) - open/close 4287 (time 1782233376.18 total 50.05 last 84.66) total: 4500 open/close in 52.32 seconds: 86.00 ops/second - /unlink 1715 (time 1782233392.47 total 10.01 last 171.39) - /unlink 3457 (time 1782233402.47 total 20.01 last 174.19) total: 4500 /unlink in 27.59 seconds: 163.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 845 (time 1782233424.53 total 10.00 last 84.48) - open/close 1648 (time 1782233434.53 total 20.01 last 80.28) - open/close 2502 (time 1782233444.54 total 30.01 last 85.35) - open/close 3404 (time 1782233454.55 total 40.02 last 90.10) - open/close 4263 (time 1782233464.55 total 50.02 last 85.89) total: 4500 open/close in 52.81 seconds: 85.22 ops/second - /unlink 1557 (time 1782233481.78 total 10.01 last 155.58) - /unlink 3003 (time 1782233491.78 total 20.01 last 144.54) - /unlink 4461 (time 1782233501.78 total 30.01 last 145.80) total: 4500 /unlink in 30.62 seconds: 146.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 775 (time 1782233517.09 total 10.00 last 77.47) - open/close 1672 (time 1782233527.09 total 20.01 last 89.63) - open/close 2563 (time 1782233537.10 total 30.02 last 89.03) - open/close 3403 (time 1782233547.10 total 40.02 last 83.99) - open/close 4260 (time 1782233557.11 total 50.03 last 85.66) total: 4500 open/close in 52.67 seconds: 85.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1583 (time 1782233573.97 total 10.00 last 158.24) - /unlink 3256 (time 1782233583.97 total 20.01 last 167.22) total: 4500 /unlink in 28.92 seconds: 155.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 959 (time 1782233607.77 total 10.01 last 95.79) - open/close 1852 (time 1782233617.78 total 20.02 last 89.25) - open/close 2720 (time 1782233627.78 total 30.02 last 86.79) - open/close 3698 (time 1782233637.78 total 40.02 last 97.74) total: 4500 open/close in 49.25 seconds: 91.36 ops/second - /unlink 1225 (time 1782233660.73 total 10.00 last 122.50) - /unlink 2910 (time 1782233670.73 total 20.00 last 168.46) - /unlink 4333 (time 1782233680.73 total 30.00 last 142.30) total: 4500 /unlink in 31.31 seconds: 143.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 732 (time 1782233696.58 total 10.01 last 73.10) - open/close 1543 (time 1782233706.59 total 20.02 last 81.04) - open/close 2466 (time 1782233716.60 total 30.03 last 92.23) - open/close 3380 (time 1782233726.60 total 40.03 last 91.38) - open/close 4262 (time 1782233736.60 total 50.03 last 88.17) total: 4500 open/close in 52.49 seconds: 85.73 ops/second - /unlink 1367 (time 1782233753.36 total 10.01 last 136.61) - /unlink 2829 (time 1782233763.36 total 20.01 last 146.19) - /unlink 4466 (time 1782233773.37 total 30.01 last 163.62) total: 4500 /unlink in 30.27 seconds: 148.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 795 (time 1782233788.16 total 10.02 last 79.32) - open/close 1577 (time 1782233798.16 total 20.02 last 78.20) - open/close 2237 (time 1782233808.18 total 30.04 last 65.89) - open/close 2936 (time 1782233818.18 total 40.05 last 69.85) - open/close 3715 (time 1782233828.20 total 50.06 last 77.80) total: 4500 open/close in 59.90 seconds: 75.12 ops/second - /unlink 1758 (time 1782233852.40 total 10.00 last 175.74) - /unlink 3405 (time 1782233862.40 total 20.01 last 164.67) total: 4500 /unlink in 27.70 seconds: 162.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 703 (time 1782233885.38 total 10.01 last 70.25) - open/close 1497 (time 1782233895.39 total 20.02 last 79.33) - open/close 2231 (time 1782233905.41 total 30.03 last 73.28) - open/close 3097 (time 1782233915.41 total 40.03 last 86.59) - open/close 3994 (time 1782233925.41 total 50.03 last 89.69) total: 4500 open/close in 55.77 seconds: 80.69 ops/second - /unlink 1625 (time 1782233945.68 total 10.00 last 162.43) - /unlink 3102 (time 1782233955.68 total 20.01 last 147.65) total: 4500 /unlink in 28.88 seconds: 155.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 958 (time 1782233978.84 total 10.01 last 95.71) - open/close 1860 (time 1782233988.85 total 20.02 last 90.08) - open/close 2782 (time 1782233998.86 total 30.03 last 92.14) - open/close 3736 (time 1782234008.86 total 40.03 last 95.38) total: 4500 open/close in 48.60 seconds: 92.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1441 (time 1782234032.16 total 10.00 last 144.09) - /unlink 2981 (time 1782234042.17 total 20.00 last 153.96) - /unlink 4165 (time 1782234052.17 total 30.01 last 118.37) total: 4500 /unlink in 32.29 seconds: 139.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 789 (time 1782234070.14 total 10.01 last 78.84) - open/close 1597 (time 1782234080.15 total 20.01 last 80.74) - open/close 2446 (time 1782234090.15 total 30.02 last 84.87) - open/close 3355 (time 1782234100.16 total 40.03 last 90.82) - open/close 4194 (time 1782234110.17 total 50.04 last 83.79) total: 4500 open/close in 54.11 seconds: 83.16 ops/second - /unlink 1402 (time 1782234129.34 total 10.00 last 140.20) - /unlink 2534 (time 1782234139.34 total 20.00 last 113.17) - /unlink 3708 (time 1782234149.35 total 30.01 last 117.34) total: 4500 /unlink in 36.47 seconds: 123.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 849 (time 1782234171.63 total 10.01 last 84.84) - open/close 1656 (time 1782234181.63 total 20.01 last 80.68) - open/close 2493 (time 1782234191.64 total 30.02 last 83.62) - open/close 3412 (time 1782234201.64 total 40.02 last 91.89) - open/close 4273 (time 1782234211.66 total 50.03 last 86.01) total: 4500 open/close in 52.57 seconds: 85.60 ops/second - /unlink 1291 (time 1782234229.37 total 10.00 last 129.09) - /unlink 2831 (time 1782234239.37 total 20.00 last 153.96) - /unlink 4396 (time 1782234249.38 total 30.01 last 156.40) total: 4500 /unlink in 30.91 seconds: 145.58 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94829 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: modules unloaded.