\n",
"

"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"with plat.ProcessPoolEvaluator(10) as evaluator:\n",
" results = plat.experiment(algorithms, problems, nfe=10000, seeds=250, evaluator=evaluator)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once the above execution has completed, we can initialize an instance of the hypervolume indicator provided by Platypus."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"hyp = plat.Hypervolume(minimum=[0, 0, 0], maximum=[1, 1, 1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can use the `calculate` function provided by Platypus to calculate all our hypervolume indicator measurements for the results from our above experiment."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"hyp_result = plat.calculate(results, hyp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, let's get the hypervolume indicator scores for all executions of NSGA-II on DTLZ1."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"hyp_result = hyp_result['NSGAII']['DTLZ1']['Hypervolume']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Calculating and Plotting the Sample Error from the Mean"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It may be tempting at this point to start generating results with other algorithms to start our comparison, however, it's important to determine a sufficient sample size before moving on. One approach is to look at the relationship between sample sizes and the Standard Error of the Mean (SEM). The formula for this is $SE_M = \\frac{s}{\\sqrt{n}}$\n",
"\n",
"Let's use the `sem` function from `scipy.stats` to calculate the $SE_M$ for each sample size made possible by our experiment above. We will incrementally append these to a list so that we can plot them later."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"SEM = []\n",
"\n",
"for sample_size in range(3,len(hyp_result)):\n",
" SEM.append(stats.sem(hyp_result[0:sample_size]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All that's left now is to plot the $SE_M$ for each incrementally ascending sample size.\n",
""
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Warning

\n", "Running the code below will take a long time to complete even if you have good hardware. To put things into perspective, you will be executing an optimisation process 250 times, per 1 test problem, per 1 algorithm. That's 250 executions of 10,000 function evaluations, totalling in at 2,500,000 function evaluations.

\n", "We are also using the `ProcessPoolEvaluator`

in Platypus to speed things up.

\n",
" \n",
" \n",
" \n",
" \n",
"

"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig = go.Figure(\n",
" data=go.Scatter(x=list(range(3,len(hyp_result))), y=SEM),\n",
" layout=dict(xaxis=dict(title='Sample Size'),yaxis=dict(title='Standard Error of the Mean'))\n",
")\n",
"\n",
"fig.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We may decide that a sufficient sample size can be selected when the $SE_M$ starts to settle below $0.05$. In this case, a sample size of $50$ can be justified."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conclusion"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this section, we have generated a large sample of results for a single algorithm on a single problem, and we have then calculated the sample error from the mean incrementally on all possible sample sizes. This has allowed us to determine a sample size that may be sufficient in the rest of our experiment. You will find different sample sizes used throughout the literature, however, you will very rarely find a clear justification for the selection. Using this approach, you can increase your confidence in the number of samples in your experiments."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"nikola": {
"category": "practical-evolutionary-algorithms",
"date": "2019-11-15 12:34:15 UTC",
"description": "",
"extra": "yes",
"link": "",
"slug": "sample-size-sufficiency",
"tags": "",
"title": "Sample Size Sufficiency",
"type": "text"
}
},
"nbformat": 4,
"nbformat_minor": 4
}