1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
|
## TestApp
The TestApp is currently being used as a dummy app by Circle CI for nightly jobs. The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. This is where Fastlane came to rescue. [Fastlane](https://fastlane.tools/) is a trendy automation tool for building and managing iOS applications. It also works seamlessly with Circle CI. We are going to leverage the `import_certificate` action, which can install developer certificates on CI machines. See `Fastfile` for more details.
For simulator build, we run unit tests as the last step of our CI workflow. Those unit tests can also be run manually via the `fastlane scan` command.
## Run Simulator Test Locally
Follow these steps if you want to run the test locally.
1. Checkout PyTorch repo including all submodules
2. Build PyTorch for ios
```
USE_COREML_DELEGATE=1 IOS_PLATFORM=SIMULATOR ./scripts/build_ios.sh
```
3. Generate on-the-fly test models
```
python test/mobile/model_test/gen_test_model.py ios-test
```
You need to install regular PyTorch on your local machine to run this script.
Check https://github.com/pytorch/pytorch/tree/master/test/mobile/model_test#diagnose-failed-test to learn more.
4. Create XCode project (for lite interpreter)
```
cd ios/TestApp/benchmark
ruby setup.rb --lite 1
```
5. Open the generated TestApp/TestApp.xcodeproj in XCode and run simulator test.
## Re-generate All Test Models
1. Make sure PyTorch (not PyTorch for iOS) is installed
See https://pytorch.org/get-started/locally/
2. Re-generate models for operator test
```
python test/mobile/model_test/gen_test_model.py ios
python test/mobile/model_test/gen_test_model.py ios-test
```
3. Re-generate Core ML model
```
cd ios/TestApp/benchmark; python coreml_backend.py
```
## Debug Test Failures
Make sure all models are generated. See https://github.com/pytorch/pytorch/tree/master/test/mobile/model_test to learn more.
There's no debug information in simulator test (project TestAppTests). You can copy the failed test code to
TestApp/TestApp/ViewController.mm and debug in the main TestApp.
### Benchmark
The benchmark folder contains two scripts that help you setup the benchmark project. The `setup.rb` does the heavy-lifting jobs of setting up the XCode project, whereas the `trace_model.py` is a Python script that you can tweak to generate your model for benchmarking. Simply follow the steps below to setup the project
1. In the PyTorch root directory, run `IOS_ARCH=arm64 ./scripts/build_ios.sh` to generate the custom build from **Master** branch
2. Navigate to the `benchmark` folder, run `python trace_model.py` to generate your model.
3. In the same directory, open `config.json`. Those are the input parameters you can tweak.
4. Again, in the same directory, run `ruby setup.rb` to setup the XCode project.
5. Open the `TestApp.xcodeproj`, you're ready to go.
The benchmark code is written in C++, you can use `UI_LOG` to visualize the log. See `benchmark.mm` for more details.
|
