1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426
|
# Table of contents
- [Overview](#overview)
- [Getting the tests to run](#running-the-tests)
- [Setting up a test environment](#setting-up-a-test-environment)
- [Running the tests](#running-the-tests)
- [Getting Azure credentials](#getting-azure-credentials)
- [Get a token with Azure Active Directory user/password](#get-a-token-with-azure-active-directory-userpassword)
- [Get a token with Active Directory application and service principal](#get-a-token-with-active-directory-application-and-service-principal)
- [Providing credentials to the tests](#providing-credentials-to-the-tests)
- [Running tests in live mode](#running-tests-in-live-mode)
- [Using the Azure Python SDK test framework](#using-the-azure-python-sdk-test-framework)
IMPORTANT NOTE: All the commands in this page assumes you have loaded the [dev_setup](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/dev_setup.md) in your currently loaded virtual environment.
# Overview
This page is to help you write tests for Azure Python SDK when these tests require Azure HTTP requests.
The Azure SDK test framework uses the [`azure-devtools`](https://pypi.python.org/pypi/azure-devtools) package,
which in turn rests upon on a HTTP recording system ([vcrpy](https://pypi.python.org/pypi/vcrpy))
that enables tests depending on network interaction
to be run offline.
In this document, we will describe:
- How to run the tests offline, using previously recorded HTTP interactions,
or online, by authenticating with Azure to record new HTTP interactions
- How to write new tests using our utility classes
# Getting the tests to run
This section describes how to run the SDK tests,
by installing dependencies into a virtual environment
and using a helper script and the [pytest](https://docs.pytest.org/en/latest/) test runner
to choose which tests to run.
## Running the tests
Azure SDK tests use [pytest](https://docs.pytest.org/en/latest/) test runner.
To run all the tests, you can use the following command:
```Shell
pytest
```
You can provide directories or individual files as positional arguments
to specify particular tests to run
rather than running the entire test suite. For example:
```Shell
pytest -s sdk/storage/azure-mgmt-storage/
pytest sdk/storage/azure-mgmt-storage/tests/test_mgmt_storage.py
```
By default, tests run in playback mode,
using recordings of HTTP interactions to simulate
requests made against Azure and allow the tests to run offline.
To run the tests in live (or "recording") mode,
you'll need to set up token-based Azure authentication.
## Getting Azure credentials
There are several ways to authenticate to Azure,
but to be able to record test HTTP interactions,
you must use an OAuth authentication method which gives you a token:
- Azure Active Directory user/password
- Active Directory application and service principal
Certificate authentication does not allow you to record HTTP queries for testing.
### Get a token with Azure Active Directory user/password. This is considered deprecated and should not be used anymore (https://docs.microsoft.com/python/azure/python-sdk-azure-authenticate?view=azure-python#mgmt-auth-legacy).
1. Connect to the [Azure Classic Portal](https://manage.windowsazure.com/) with your admin account.
2. Create a user in your default AAD https://azure.microsoft.com/documentation/articles/active-directory-create-users/
**You must NOT activate Multi-Factor Authentication!**
3. Go to Settings - Administrators.
4. Click on *Add* and enter the email of the new user.
Check the checkbox of the subscription you want to test with this user.
5. Login to Azure Portal with this new user to change the temporary password to a new one.
You will not be able to use the temporary password for OAuth login.
You are now able to log in Python using OAuth. You can test with this code:
```python
from azure.common.credentials import UserPassCredentials
credentials = UserPassCredentials(
'user@domain.com', # Your new user
'my_smart_password', # Your password
)
```
### Get a token with Active Directory application and service principal
Follow this detailed tutorial to set up an Active Directory application and service principal:
https://azure.microsoft.com/documentation/articles/resource-group-create-service-principal-portal/
To use the credentials from Python,
you need the application ID (a.k.a. client ID),
authentication key (a.k.a. client secret),
tenant ID and subscription ID from the Azure portal for use in the next step.
[This section of the above tutorial](https://docs.microsoft.com/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key)
describes where to find them
(besides the subscription ID,
which is in the "Overview" section of the "Subscriptions" blade.)
You are now able to log in from Python using OAuth.
You can test with this code:
```python
from azure.common.credentials import ServicePrincipalCredentials
credentials = ServicePrincipalCredentials(
client_id = 'ABCDEFGH-1234-ABCD-1234-ABCDEFGHIJKL',
secret = 'XXXXXXXXXXXXXXXXXXXXXXXX',
tenant = 'ABCDEFGH-1234-ABCD-1234-ABCDEFGHIJKL'
)
```
## Providing credentials to the tests
When you run tests in playback mode,
they use a fake credentials file,
located at `tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py`,
to simulate authenticating with Azure.
In live mode, you need to use actual credentials
like those you obtained in the previous section.
To enable the tests to use them,
make a copy of the `mgmt_settings_fake.py` file in the same location,
and rename it `mgmt_settings_real.py`.
Then make the following changes:
* Change the value of the `SUBSCRIPTION_ID` constant to your subscription ID.
(If you don't have it,
you can find it in the "Overview" section of the "Subscriptions" blade
in the [Azure portal](https://portal.azure.com/).)
* Change the `get_credentials()` function to construct and return
a `UserPassCredentials` or `ServicePrincipalCredentials` object
such as you constructed in the samples in the previous section.
(Don't forget to make sure the necessary imports are present as well!)
**Important: `mgmt_settings_real.py` should not be committed since it contains your actual credentials! To prevent this, it is included in `.gitignore`.**
## Running tests in live mode
To configure the tests to run in live mode, you have two options:
* Set the environment variable `AZURE_TEST_RUN_LIVE` to "true" or "yes".
If you want to go back to playback mode you can either unset it entirely
or set it to "false" or "no".
* Create a `testsettings_local.cfg` file in the same directory as
`mgmt_settings_real.py`. It should look like the following:
```
live-mode: true
```
To go back to playback mode using the config file,
change the "true" to "false" or delete the file.
(`testsettings_local.cfg` is listed in `.gitignore`
and not present in the repo; if it's missing,
the tests default to playback mode.)
Now you can run tests using the same method described in
[Running the tests](#running-the-tests).
You would be well-advised to specify a limited number of tests to run.
Running every existing test in live mode will take a very long time
and produce a great deal of changes to recording files in your Git repository.
# Writing new tests
SDK tests are based on the `scenario_tests` subpackage of the
[`azure-devtools`](https://pypi.python.org/pypi/azure-devtools) package.
`scenario_tests` is a general, mostly abstract framework
providing several features useful for the SDK tests, for example:
* HTTP interaction recording and playback using [vcrpy](https://pypi.python.org/pypi/vcrpy)
* Creation and cleanup of helper resources, such as resource groups,
for tests that need them in order to test other services
* Processors for modifying requests and responses when writing or reading recordings
(for example, to avoid recording credential information)
* Patches for overriding functions and methods that don't work well with tests
(such as long-running operations)
Code in the [`azure-sdk-tools/devtools_testutils`](https://github.com/Azure/azure-sdk-for-python/tree/master/tools/azure-sdk-tools/devtools_testutils) directory
provides concrete implementations of the features provided in `scenario_tests`
that are oriented around use in SDK testing
and that you can use directly in your unit tests.
## Test structure
New tests should be located alongside the packages containing the code they test.
For example, the tests for `azure-mgmt-media` are in `azure-mgmt-media/tests`.
Each test folder also has a `recordings` subfolder containing one .yaml recording file
for each test that has HTTP interactions to record.
There are also legacy tests in the following three locations:
* azure-servicebus/tests
* azure-servicemanagement-legacy/tests
For more information about legacy tests, see [Legacy tests](https://github.com/Azure/azure-sdk-for-python/wiki/Legacy-tests).
## Using the Azure Python SDK test framework
This section will demonstrate writing tests using `devtools_testutils`
with a few increasingly sophisticated examples
to show how to use some of the features of the underlying test frameworks.
### Example 1: Basic Azure service interaction and recording
```python
from azure.mgmt.resource import ResourceManagementClient
from devtools_testutils import AzureMgmtTestCase
class ExampleResourceGroupTestCase(AzureMgmtTestCase):
def setUp(self):
super(ExampleResourceGroupTestCase, self).setUp()
self.client = self.create_mgmt_client(ResourceManagementClient)
def test_create_resource_group(self):
test_group_name = self.get_resource_name('testgroup')
group = self.client.resource_groups.create_or_update(
test_group_name,
{'location': 'westus'}
)
self.assertEqual(group.name, test_group_name)
self.client.resource_groups.delete(group.name).wait()
```
This simple test creates a resource group and checks that its name
is assigned correctly.
Notes:
1. This test inherits all necessary behavior for HTTP recording and playback
described previously in this document
from its `AzureMgmtTestCase` superclass.
You don't need to do anything special to implement it.
2. The `get_resource_name()` helper method of `AzureMgmtTestCase`
creates a pseudorandom name based on the parameter
and the names of the test file and method.
This ensures that the name generated is the same for each run
of the same test, thereby ensuring reproducability,
but prevents name collisions if the tests are run live
and the same parameter is used from several different tests.
3. The `create_mgmt_client()` helper method of `AzureMgmtTestCase`
creates a client object using the credentials
from `mgmt_settings_fake.py` or `mgmt_settings_real.py` as appropriate,
with some checks to make sure it's created successfully
and cause the unit test to fail if not.
You should use it for any clients you create.
4. Note that this test cleans up the resource group it creates!
If you create resources yourself as part of the test,
make sure to delete them afterwards.
But if you need something like a resource group
as a prerequisite for what you're actually trying to test,
you should use a "preparer" as demonstrated in the following two examples.
Preparers will create and clean up helper resources for you.
### Example 2: Basic preparer usage
```python
from azure.mgmt.sql import SqlManagementClient
from devtools_testutils import AzureMgmtTestCase, ResourceGroupPreparer
class ExampleSqlServerTestCase(AzureMgmtTestCase):
def setUp(self):
super(ExampleSqlServerTestCase, self).setUp()
self.client = self.create_mgmt_client(SqlManagementClient)
@ResourceGroupPreparer()
def test_create_sql_server(self, resource_group, location):
test_server_name = self.get_resource_name('testsqlserver')
server_creation = self.client.servers.create_or_update(
resource_group.name,
test_server_name,
{
'location': location,
'version': '12.0',
'administrator_login': 'mysecretname',
'administrator_login_password': 'HusH_Sec4et'
}
)
server = server_creation.result()
self.assertEqual(server.name, test_server_name)
```
This test creates a SQL server and confirms that its name is set correctly.
Because a SQL server must be created in a resource group,
the test uses a `ResourceGroupPreparer` to create a group for use in the test.
Preparers are [decorators](https://www.python.org/dev/peps/pep-0318/)
that "wrap" a test method,
transparently replacing it with another function that has some additional functionality
before and after it's run.
For example, the `@ResourceGroupPreparer` decorator adds the following to the wrapped method:
* creates a resource group
* inspects the argument signature of the wrapped method
and passes in information about the created resource group
if appropriately-named parameters
(here, `resource_group` and `location`) are present
* deletes the resource group after the test is run
Notes:
1. HTTP interactions undertaken by preparers
to create and delete the prepared resource
are not recorded or played back,
as they're not part of what the test is testing.
2. If the test is run in playback mode,
the `resource_group` parameter will be a simple `FakeResource` object
with a pseudorandom `name` attribute and a blank `id` attribute.
If you need a more sophisticated fake object, see the next example.
3. Why not use a preparer in Example 1, above?
Preparers are only for *auxiliary* resources
that aren't part of the main focus of the test.
In example 1, we want to test the actual creation and naming
of the resource group, so those operations are part of the test.
By contrast, in example 2, the subject of the test
is the SQL server management operations;
the resource group is just a prerequisite for those operations.
We only want this test to fail if something is wrong with the SQL server creation.
If there's something wrong with the resource group creation,
there should be a dedicated test for that.
### Example 3: More complicated preparer usage
```python
from azure.mgmt.media import MediaServicesManagementClient
from devtools_testutils import (
AzureMgmtTestCase, ResourceGroupPreparer, StorageAccountPreparer, FakeResource
)
FAKE_STORAGE_ID = 'STORAGE-FAKE-ID'
FAKE_STORAGE = FakeResource(name='teststorage', id=FAKE_STORAGE_ID)
class ExampleMediaServiceTestCase(AzureMgmtTestCase):
def setUp(self):
super(ExampleMediaServiceTestCase, self).setUp()
self.client = self.create_mgmt_client(MediaServicesManagementClient)
@ResourceGroupPreparer(parameter_name='group')
@StorageAccountPreparer(playback_fake_resource=FAKE_STORAGE,
name_prefix='testmedia',
resource_group_parameter_name='group')
def test_create_media_service(self, group, location, storage_account):
test_media_name = self.get_resource_name('pymediatest')
media_obj = self.client.media_service.create(
group.name,
test_media_name,
{
'location': location,
'storage_accounts': [{
'id': storage_account.id,
'is_primary': True,
}]
}
)
self.assertEqual(media_obj.name, test_media_name)
```
This test creates a media service and confirms that its name is set correctly.
Notes:
1. Here, we want to test creation of a media service,
which requires a storage account.
We want to use a preparer for this,
but creation of a storage account itself needs a resource group.
So we need both a `ResourceGroupPreparer` and a `StorageAccountPreparer`,
in that order.
2. Both preparers are customized.
We pass a `parameter_name` keyword argument of `group` to `ResourceGroupPreparer`,
and as a result the resource group is passed into the test method
through the `group` parameter (rather than the default `resource_group`).
Then, because `StorageAccountPreparer` needs a resource group,
we need to let it know about the modified parameter name.
We do so with the `resource_group_parameter_name` argument.
Finally, we pass a `name_prefix` to `StorageAccountPreparer`.
The names it generates by default include the fully qualified test name,
and so tend to be longer than is allowed for storage accounts.
You'll probably always need to use `name_prefix` with `StorageAccountPreparer`.
3. We want to ensure that the group retrieved by `get_properties`
has a `kind` of `BlobStorage`.
We create a `FakeStorageAccount` object with that attribute
and pass it to `StorageAccountPreparer`,
and also pass the `kind` keyword argument to `StorageAccountPreparer`
so that it will be passed through when a storage account is prepared for real.
4. Similarly to how a resource group parameter is added by `ResourceGroupPreparer`,
`StorageAccountPreparer` passes the model object for the created storage account
as the `storage_account` parameter, and that parameter's name can be customized.
`StorageAccountPreparer` also creates an account access key
and passes it into the test method through a parameter whose name is formed
by appending `_key` to the name of the parameter for the account itself.
### Example 4: Different endpoint than public Azure (China, Dogfood, etc.)
```python
from azure.mgmt.sql import SqlManagementClient
from devtools_testutils import AzureMgmtTestCase, ResourceGroupPreparer
_CUSTOM_ENDPOINT = "https://api-dogfood.resources.windows-int.net/"
class ExampleSqlServerTestCase(AzureMgmtTestCase):
def setUp(self):
super(ExampleSqlServerTestCase, self).setUp()
self.client = self.create_mgmt_client(
SqlManagementClient,
base_url=_CUSTOM_ENDPOINT
)
@ResourceGroupPreparer(client_kwargs={'base_url':_CUSTOM_ENDPOINT})
def test_create_sql_server(self, resource_group, location):
test_server_name = self.get_resource_name('testsqlserver')
server_creation = self.client.servers.create_or_update(
resource_group.name,
test_server_name,
{
'location': location,
'version': '12.0',
'administrator_login': 'mysecretname',
'administrator_login_password': 'HusH_Sec4et'
}
)
server = server_creation.result()
self.assertEqual(server.name, test_server_name)
```
|