path: root/railties/doc/guides/benchmarking_and_profiling/gameplan.txt



== Get Yourself a Game Plan ==

You end up dealing with a large amount of data whenever you profile an application. It's crucial to use a rigorous approach to analyzing your application's performance else fail miserably in a vortex of numbers. This leads us to -

=== The Analysis Process ===

I’m going to give an example methodology for conducting your benchmarking and profiling on an application. It is based on your typical scientific method. 

For something as complex as Benchmarking you need to take any methodology with a grain of salt but there are some basic strictures that you can depend on.

Formulate a question you need to answer which is simple, tests the smallest measurable thing possible, and is exact. This is typically the hardest part of the experiment. From there some steps that you should follow are. 

* Develop a set of variables and processes to measure in order to answer this question!
* Profile based on the question and variables. Key problems to avoid when designing this experiment are:
	- Confounding: Test one thing at a time, keep everything the same so you don't poison the data with uncontrolled processes. 
	- Cross Contamination: Make sure that runs from one test do not harm the other tests.
	- Steady States: If you’re testing long running process. You must take the ramp up time and performance hit into your initial measurements.
	- Sampling Error: Data should perform have a steady variance or range. If you get wild swings or sudden spikes, etc. then you must either account for the reason why or you have a sampling error.
	- Measurement Error: Aka Human error, always go through your calculations at least twice to make sure there are no mathematical errors. .
* Do a small run of the experiment to verify the design.
* Use the small run to determine a proper sample size.
* Run the test.
* Perform the analysis on the results and determine where to go from there.

Note: Even though we are using the typical scientific method; developing a hypothesis is not always useful in terms of profiling.
== Get Yourself a Game Plan ==

You end up dealing with a large amount of data whenever you profile an application. It's crucial to use a rigorous approach to analyzing your application's performance else fail miserably in a vortex of numbers. This leads us to -

=== The Analysis Process ===

I’m going to give an example methodology for conducting your benchmarking and profiling on an application. It is based on your typical scientific method. 

For something as complex as Benchmarking you need to take any methodology with a grain of salt but there are some basic strictures that you can depend on.

Formulate a question you need to answer which is simple, tests the smallest measurable thing possible, and is exact. This is typically the hardest part of the experiment. From there some steps that you should follow are. 

* Develop a set of variables and processes to measure in order to answer this question!
* Profile based on the question and variables. Key problems to avoid when designing this experiment are:
	- Confounding: Test one thing at a time, keep everything the same so you don't poison the data with uncontrolled processes. 
	- Cross Contamination: Make sure that runs from one test do not harm the other tests.
	- Steady States: If you’re testing long running process. You must take the ramp up time and performance hit into your initial measurements.
	- Sampling Error: Data should perform have a steady variance or range. If you get wild swings or sudden spikes, etc. then you must either account for the reason why or you have a sampling error.
	- Measurement Error: Aka Human error, always go through your calculations at least twice to make sure there are no mathematical errors. .
* Do a small run of the experiment to verify the design.
* Use the small run to determine a proper sample size.
* Run the test.
* Perform the analysis on the results and determine where to go from there.

Note: Even though we are using the typical scientific method; developing a hypothesis is not always useful in terms of profiling.