Skip to content
Snippets Groups Projects
Commit 851033fd authored by irl's avatar irl :keyboard:
Browse files

metrics/cloud: metrics-vpc

parent d7bb11e7
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
metrics/ops/metrics-cloud.html
+ 117
− 60
View file @ 851033fd
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2020-04-01 Wed 11:07 -->
<!-- 2020-04-01 Wed 11:22 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>metrics-cloud: Scripts for orchestrating Tor Metrics services</title>
......@@ -234,40 +234,40 @@ for the JavaScript code in this tag.
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#org185431b">1. <span class="done DONE">DONE</span> Synopsis</a></li>
<li><a href="#org2fff687">2. <span class="done DONE">DONE</span> Usage of AWS for Tor Metrics Development</a>
<li><a href="#orgaefea1a">1. <span class="done DONE">DONE</span> Synopsis</a></li>
<li><a href="#org0f4787e">2. <span class="done DONE">DONE</span> Usage of AWS for Tor Metrics Development</a>
<ul>
<li><a href="#orgf211360">2.1. <span class="done DONE">DONE</span> CloudFormation Templates</a>
<li><a href="#orge02eb3f">2.1. <span class="done DONE">DONE</span> CloudFormation Templates</a>
<ul>
<li><a href="#org60761d6">2.1.1. <span class="done DONE">DONE</span> Quickstart: Deploying a template</a></li>
<li><a href="#org90ac834">2.1.2. <span class="done DONE">DONE</span> SSH Key Selection</a></li>
<li><a href="#org33f903f">2.1.1. <span class="done DONE">DONE</span> Quickstart: Deploying a template</a></li>
<li><a href="#orgb8608ce">2.1.2. <span class="done DONE">DONE</span> SSH Key Selection</a></li>
</ul>
</li>
<li><a href="#org393c195">2.2. <span class="done DONE">DONE</span> Development DNS</a></li>
<li><a href="#org240eb03">2.3. <span class="todo TODO">TODO</span> The Templates</a>
<li><a href="#org3a60b6a">2.2. <span class="todo TODO">TODO</span> The Templates</a>
<ul>
<li><a href="#org6696453">2.3.1. <span class="done DONE">DONE</span> <code>billing-alerts</code></a></li>
<li><a href="#orgfb7a921">2.3.2. <span class="todo TODO">TODO</span> <code>metrics-vpc</code></a></li>
<li><a href="#org8f84302">2.3.3. <span class="todo TODO">TODO</span> Typical Dev/Testing Stacks</a></li>
<li><a href="#orgb0dbe50">2.2.1. <span class="todo TODO">TODO</span> Naming conventions</a></li>
<li><a href="#orga6d4cbf">2.2.2. <span class="done DONE">DONE</span> <code>billing-alerts</code></a></li>
<li><a href="#org18a75e3">2.2.3. <span class="todo TODO">TODO</span> <code>metrics-vpc</code></a></li>
<li><a href="#org72e9c82">2.2.4. <span class="todo TODO">TODO</span> Typical Dev/Testing Stacks</a></li>
</ul>
</li>
<li><a href="#org132a620">2.4. <span class="todo TODO">TODO</span> Linting</a></li>
<li><a href="#org38d181d">2.3. <span class="todo TODO">TODO</span> Linting</a></li>
</ul>
</li>
<li><a href="#orgf3e8342">3. <span class="todo TODO">TODO</span> Ansible Playbooks</a>
<li><a href="#org6705221">3. <span class="todo TODO">TODO</span> Ansible Playbooks</a>
<ul>
<li><a href="#orgb759208">3.1. <span class="todo TODO">TODO</span> Inventory and site.yml</a></li>
<li><a href="#org83b55cb">3.2. <span class="todo TODO">TODO</span> <code>metrics-common</code></a></li>
<li><a href="#orgf142a79">3.3. <span class="todo TODO">TODO</span> <code>*-sys</code> roles</a></li>
<li><a href="#orgaf323c9">3.4. <span class="todo TODO">TODO</span> service roles</a></li>
<li><a href="#org9fb50b0">3.1. <span class="todo TODO">TODO</span> Inventory and site.yml</a></li>
<li><a href="#org1259bed">3.2. <span class="todo TODO">TODO</span> <code>metrics-common</code></a></li>
<li><a href="#org1ed4393">3.3. <span class="todo TODO">TODO</span> <code>*-sys</code> roles</a></li>
<li><a href="#org40dddd8">3.4. <span class="todo TODO">TODO</span> service roles</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div id="outline-container-org185431b" class="outline-2">
<h2 id="org185431b"><span class="section-number-2">1</span> <span class="done DONE">DONE</span> Synopsis</h2>
<div id="outline-container-orgaefea1a" class="outline-2">
<h2 id="orgaefea1a"><span class="section-number-2">1</span> <span class="done DONE">DONE</span> Synopsis</h2>
<div class="outline-text-2" id="text-1">
<p>
The metrics-cloud framework aims to enable:
......@@ -296,8 +296,8 @@ to both environments.
</div>
</div>
<div id="outline-container-org2fff687" class="outline-2">
<h2 id="org2fff687"><span class="section-number-2">2</span> <span class="done DONE">DONE</span> Usage of AWS for Tor Metrics Development</h2>
<div id="outline-container-org0f4787e" class="outline-2">
<h2 id="org0f4787e"><span class="section-number-2">2</span> <span class="done DONE">DONE</span> Usage of AWS for Tor Metrics Development</h2>
<div class="outline-text-2" id="text-2">
<p>
Each member of the Tor Metrics team has a standing allowance of 100USD/month for development using AWS. In practice,
......@@ -307,8 +307,8 @@ rapid creation, provisioning and destruction should help with this.
</p>
</div>
<div id="outline-container-orgf211360" class="outline-3">
<h3 id="orgf211360"><span class="section-number-3">2.1</span> <span class="done DONE">DONE</span> CloudFormation Templates</h3>
<div id="outline-container-orge02eb3f" class="outline-3">
<h3 id="orge02eb3f"><span class="section-number-3">2.1</span> <span class="done DONE">DONE</span> CloudFormation Templates</h3>
<div class="outline-text-3" id="text-2-1">
<p>
CloudFormation is an AWS service allowing the definition of <i>stacks</i>. These stacks describe a series of AWS services
......@@ -332,8 +332,8 @@ Documentation for CloudFormation, including an API reference, can be found at: <
</p>
</div>
<div id="outline-container-org60761d6" class="outline-4">
<h4 id="org60761d6"><span class="section-number-4">2.1.1</span> <span class="done DONE">DONE</span> Quickstart: Deploying a template</h4>
<div id="outline-container-org33f903f" class="outline-4">
<h4 id="org33f903f"><span class="section-number-4">2.1.1</span> <span class="done DONE">DONE</span> Quickstart: Deploying a template</h4>
<div class="outline-text-4" id="text-2-1-1">
<p>
Each template begins with comments with any relevant notes about the template, and a deployment command that will upload
......@@ -358,8 +358,8 @@ the <a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1
</div>
</div>
<div id="outline-container-org90ac834" class="outline-4">
<h4 id="org90ac834"><span class="section-number-4">2.1.2</span> <span class="done DONE">DONE</span> SSH Key Selection</h4>
<div id="outline-container-orgb8608ce" class="outline-4">
<h4 id="orgb8608ce"><span class="section-number-4">2.1.2</span> <span class="done DONE">DONE</span> SSH Key Selection</h4>
<div class="outline-text-4" id="text-2-1-2">
<p>
The <a href="https://gitweb.torproject.org/metrics-cloud.git/tree/cloudformation/identify_user.sh">identify_user.sh</a> script prints out the name of the SSH public key to be used based on either:
......@@ -385,9 +385,78 @@ SSH keys are managed through the <a href="https://console.aws.amazon.com/ec2/v2/
</div>
</div>
<div id="outline-container-org393c195" class="outline-3">
<h3 id="org393c195"><span class="section-number-3">2.2</span> <span class="done DONE">DONE</span> Development DNS</h3>
<div id="outline-container-org3a60b6a" class="outline-3">
<h3 id="org3a60b6a"><span class="section-number-3">2.2</span> <span class="todo TODO">TODO</span> The Templates</h3>
<div class="outline-text-3" id="text-2-2">
</div>
<div id="outline-container-orgb0dbe50" class="outline-4">
<h4 id="orgb0dbe50"><span class="section-number-4">2.2.1</span> <span class="todo TODO">TODO</span> Naming conventions</h4>
</div>
<div id="outline-container-orga6d4cbf" class="outline-4">
<h4 id="orga6d4cbf"><span class="section-number-4">2.2.2</span> <span class="done DONE">DONE</span> <code>billing-alerts</code></h4>
<div class="outline-text-4" id="text-2-2-2">
<p>
The <a href="https://gitweb.torproject.org/metrics-cloud.git/tree/cloudformation/billing-alerts.yml"><code>billing-alerts</code> template</a> sends notifications to the subscribed individuals whenever the predicted spend for the month will be
over 50USD. Email addresses can be added here if other people should be notified too.
</p>
</div>
</div>
<div id="outline-container-org18a75e3" class="outline-4">
<h4 id="org18a75e3"><span class="section-number-4">2.2.3</span> <span class="todo TODO">TODO</span> <code>metrics-vpc</code></h4>
<div class="outline-text-4" id="text-2-2-3">
<p>
The <a href="https://gitweb.torproject.org/metrics-cloud.git/tree/cloudformation/metrics-vpc.yml"><code>metrics-vpc</code> template</a> contains shared resources for Tor Metrics development templates. This includes:
</p>
</div>
<ol class="org-ol">
<li><a id="orga95cb3e"></a>MetricsVPC and MetricsSubnet<br />
<div class="outline-text-5" id="text-2-2-3-1">
<p>
The subnet should be referenced by any resource that requires it. Use of the default VPC should be avoided as we
share the AWS account with other Tor teams.
</p>
<p>
For example, to create an EC2 instance:
</p>
<div class="org-src-container">
<pre class="src src-yaml">Instance:
Type: AWS::EC2::Instance
Properties:
AvailabilityZone: !Select [ 0, !GetAZs ]
ImageId: ami-01db78123b2b99496
InstanceType: t2.large
SubnetId:
Fn::ImportValue: 'MetricsSubnet'
KeyName: !Ref myKeyPair
SecurityGroupIds:
- Fn::ImportValue: 'MetricsInternetSecurityGroup'
- Fn::ImportValue: 'MetricsPingableSecurityGroup'
- Fn::ImportValue: 'MetricsHTTPASecurityGroup'
</pre>
</div>
<p>
Note also that the availability zone is not hardcoded to allow for portability between regions if we ever want that.
</p>
</div>
</li>
<li><a id="orgdab3066"></a>Various security groups<br />
<div class="outline-text-5" id="text-2-2-3-2">
<p>
The EC2 example above uses some of the security groups from the <code>metrics-vpc</code> template. Refer to the template source
for details on each group's rules.
</p>
</div>
</li>
<li><a id="orgb67d0dd"></a>The development DNS zone<br />
<div class="outline-text-5" id="text-2-2-3-3">
<p>
Often services require TLS certificates, or require DNS names for other reasons. To facilitate this, a zone is hosted
using Route53 allowing for DNS records to be created in CloudFormation templates. This zone is:
......@@ -412,63 +481,51 @@ As an example, creating an A record for an EC2 instance with the subdomain of th
</div>
<p>
These domain names should <b>never</b> appear on anything user facing and are for <b>development purposes only</b>.
Q: <i>Can we use the MetricsDevZone export from <code>metrics-vpc</code> instead of explicitly defining the zone name every time?</i>
</p>
</div>
</div>
<div id="outline-container-org240eb03" class="outline-3">
<h3 id="org240eb03"><span class="section-number-3">2.3</span> <span class="todo TODO">TODO</span> The Templates</h3>
<div class="outline-text-3" id="text-2-3">
</div>
<div id="outline-container-org6696453" class="outline-4">
<h4 id="org6696453"><span class="section-number-4">2.3.1</span> <span class="done DONE">DONE</span> <code>billing-alerts</code></h4>
<div class="outline-text-4" id="text-2-3-1">
<p>
This template sends notifications to the subscribed individuals whenever the predicted spend for the month will be
over 50USD. Email addresses can be added here if other people should be notified too.
These domain names should <b>never</b> appear on anything user facing and are for <b>development purposes only</b>.
</p>
</div>
</li>
</ol>
</div>
<div id="outline-container-orgfb7a921" class="outline-4">
<h4 id="orgfb7a921"><span class="section-number-4">2.3.2</span> <span class="todo TODO">TODO</span> <code>metrics-vpc</code></h4>
</div>
<div id="outline-container-org8f84302" class="outline-4">
<h4 id="org8f84302"><span class="section-number-4">2.3.3</span> <span class="todo TODO">TODO</span> Typical Dev/Testing Stacks</h4>
<div id="outline-container-org72e9c82" class="outline-4">
<h4 id="org72e9c82"><span class="section-number-4">2.2.4</span> <span class="todo TODO">TODO</span> Typical Dev/Testing Stacks</h4>
</div>
</div>
<div id="outline-container-org132a620" class="outline-3">
<h3 id="org132a620"><span class="section-number-3">2.4</span> <span class="todo TODO">TODO</span> Linting</h3>
<div id="outline-container-org38d181d" class="outline-3">
<h3 id="org38d181d"><span class="section-number-3">2.3</span> <span class="todo TODO">TODO</span> Linting</h3>
</div>
</div>
<div id="outline-container-orgf3e8342" class="outline-2">
<h2 id="orgf3e8342"><span class="section-number-2">3</span> <span class="todo TODO">TODO</span> Ansible Playbooks</h2>
<div id="outline-container-org6705221" class="outline-2">
<h2 id="org6705221"><span class="section-number-2">3</span> <span class="todo TODO">TODO</span> Ansible Playbooks</h2>
<div class="outline-text-2" id="text-3">
</div>
<div id="outline-container-orgb759208" class="outline-3">
<h3 id="orgb759208"><span class="section-number-3">3.1</span> <span class="todo TODO">TODO</span> Inventory and site.yml</h3>
<div id="outline-container-org9fb50b0" class="outline-3">
<h3 id="org9fb50b0"><span class="section-number-3">3.1</span> <span class="todo TODO">TODO</span> Inventory and site.yml</h3>
</div>
<div id="outline-container-org83b55cb" class="outline-3">
<h3 id="org83b55cb"><span class="section-number-3">3.2</span> <span class="todo TODO">TODO</span> <code>metrics-common</code></h3>
<div id="outline-container-org1259bed" class="outline-3">
<h3 id="org1259bed"><span class="section-number-3">3.2</span> <span class="todo TODO">TODO</span> <code>metrics-common</code></h3>
</div>
<div id="outline-container-orgf142a79" class="outline-3">
<h3 id="orgf142a79"><span class="section-number-3">3.3</span> <span class="todo TODO">TODO</span> <code>*-sys</code> roles</h3>
<div id="outline-container-org1ed4393" class="outline-3">
<h3 id="org1ed4393"><span class="section-number-3">3.3</span> <span class="todo TODO">TODO</span> <code>*-sys</code> roles</h3>
</div>
<div id="outline-container-orgaf323c9" class="outline-3">
<h3 id="orgaf323c9"><span class="section-number-3">3.4</span> <span class="todo TODO">TODO</span> service roles</h3>
<div id="outline-container-org40dddd8" class="outline-3">
<h3 id="org40dddd8"><span class="section-number-3">3.4</span> <span class="todo TODO">TODO</span> service roles</h3>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="author">Author: Iain Learmonth</p>
<p class="date">Created: 2020-04-01 Wed 11:07</p>
<p class="date">Created: 2020-04-01 Wed 11:22</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment