The STochastic OPTimization library (StOpt) aims at providing tools in C++ for solving
some stochastic optimization problems encountered in finance or in the industry.
A python binding is available for some C++ objects provided permitting to easily solve an
optimization problem by regression or tree methods. 
The possibility to create a R package for the regressors is also provided for linux and
mac users.
Different methods are available :
- dynamic programming methods based on Monte Carlo  with regressions (global, local ,
kernel and  sparse regressors), for underlying states following
  some uncontrolled Stochastic Differential Equations (python binding provided).
  Transition problems can be solved by:
  - discretization of commands,
  - resolution of some LP and then the Bellman values are approximated by cuts
- dynamic programming with a representation of uncertainties with a tree :
  transition problems are here solved by
  - some discretizations of the commands,
  - resolution of LP with cut representation of the Bellman values.
- Semi-Lagrangian methods for Hamilton Jacobi Bellman general equations for underlying
states following some controlled  Stochastic Differential Equations (C++ only)
- Stochastic Dual Dynamic Programming methods to deal with stochastic stocks management
problems in high dimension. A SDDP module in python is provided.
  To use this module, the transitional optimization problem has to written in C++ and
  mapped to python (examples provided).
  Uncertainties can be given by Monte Carlo and can be represented by a state with a finite
  number of values (tree).
- Some branching nesting methods to solve very high dimensional non linear PDEs and
  some appearing in HJB problems.
Besides some methods are provided to solve by Monte Carlo some problems where the
underlying stochastic state is controlled.
For each method, a framework is provided to optimize the problem and then simulate
it out of the sample using the optimal commands previously calculated.
Parallelization methods based on OpenMP and MPI are provided in this framework
permitting to solve high dimensional problems on clusters.
The library should be flexible enough to be used at different levels depending on the
user's willingness.

############################################################################################

GENERAL DOCUMENTATION AVAILABLE AT :

https://www.researchgate.net/project/The-STochastic-OPTimization-library-https-gitlabcom-stochastic-control-StOpt

or pick up latest version in :

https://hal.archives-ouvertes.fr/hal-01361291

############################################################################################

Please, while using the library for some research purpose, please cite:

@article{gevret2018stochastic,
  title={STochastic OPTimization library in C++},
  author={Gevret, Hugo and Langren{\'e}, Nicolas and Lelong, Jerome and Warin, Xavier and Maheshwari, Aditya},
  year={2018}
}

###########################################################################################

Licence : The library is published under the GNU LESSER GENERAL PUBLIC LICENSE
(see LICENCE.txt, COPYING, COPYING.LESSER files)

###########################################################################################

News (2022/02) : from V5.0, MPI interface changes permitting to have two levels of MPI parallelization in a single software. 


###########################################################################################

The library can be built on Linux/Mac/Windows using some recent compilers.
It has been tested with
- gcc, clang, intel icc, intel DPC++ on linux,
- clang on mac
- visual studio  on windows (VS2019, VS2022)
- intel DPC++ on windows

############################################################################################

############################################################################################

##################
## Prerequisite ##
##################

StOpt library uses some external library.
One is included  in the package so do no need any installation :
- geners library  for generic serialization for C++ : http://geners.hepforge.org/

The following libraries need to be installed :
- zlib, (geners serialization, already present in linux)
- bzip2, (geners serialization, already present in linux)
- eigen library  version 3.4 or above  : http://http://eigen.tuxfamily.org/
- pybind library https://github.com/pybind/pybind11
- boost library : http://www.boost.org/ 
- python 3  with numpy package. Scipy package needed for test cases.
Mpi4py package needed for test with mpi. 
Other libraries are optional :
- mpi  http://www.open-mpi.org/ for MPI calculation, MS mpi on windows
- COIN OSI, COIN CLP, COIN UTILS  only for SDDP test  (http://www.coin-or.org/)

StOpt relies on cmake for compilation http://www.cmake.org/ and permits to generate
projects (Kdevelopp or Visual Studio for example).

##################################
## Important for multithreading ##
##################################

StOpt uses multithread : so OMP_NUM_THREADS environment variable has to be set to
a number of threads to use.


##################################
# Important to use python       ##
##################################

Python test case are located in StOpt/test/python.
To use python the PYTHONPATH and LD_LIBRARY_PATH should be updated to include the  directory
containing the StOpt (.so, .pyd)  library (default is the  lib directory where the library is built).
If the  build is achieved in /home/ME/GIT_REPOS/BUILD_RELEASE_GCC then under linux and MacOS:

```
LD_LIBRARY_PATH=/home/ME/GIT_REPOS/BUILD_RELEASE_GCC/lib/:$LD_LIBRARY_PATH
PYTHONPATH=/home/ME/GIT_REPOS/BUILD_RELEASE_GCC/lib:$PYTHONPATH
```

####################################
## INSTALLATION, TESTS  LINUX/MAC ##
####################################
####################################

Install package for zlib, bzip2, python, scipy (http://www.scipy.org/),  eigen,  boost
(including  boost mpi if necessary), cmake
and cmake gui (ccmake) using your favorite package manager (apt, snap, yaourt, .. on linux, homebrew on Mac OS)
Go to StOpt directory (containing directories  geners-1.10.0, StOpt ...)

```
cd ..  
mkdir BUILD  
cd BUILD
cmake ../StOpt
cmake ../StOpt
```


(Rerun cmake so that the python libs can be found..)
If packages for libraries are not in usual place, use ccmake (ccmake ../StOpt)
to define included directories and libraries needed.
In cmake gui define :
- if  python bindings should be compiled (default no)
- if the library should use mpi (default true)
- if the library should compile sddp test (default false (COIN CLP needed))
This gui can be called by

```
ccmake ../StOpt
make
```


Test cases are run by:
```
ctest
```

A package can be build using cpack :
- for debian :
  cpack -G DEB
- for Red Hat :
  cpack -G RPM
-  a GZIP compression :
  cpack -G TGZ

Installation in /usr/local

```
make install
```

For installation in given directory /home/toto/test
(installation will be achieved in /home/toto/test/usr/local)

```
make DESTDIR=/home/toto/test install
```


It is also possible to have installation in /home/toto/test such that the
directory tree looks like /home/toto/test/lib, /home/toto/test/include...

```
cmake -DCMAKE_INSTALL_PREFIX=/home/toto/test ../StOpt
make install
```

##########################################################################
##    Python test                                                        #
##########################################################################

To run the python tests  if library compiled with python (don't forget to update
PYTHONPATH in your .bashrc):

```
cd ../StOpt/test/python
python -m unittest discover
```

In StOpt/test/python/functional  python version of C++ test cases are available.

Two kind of mappings are available.
- A first mapping is achieved at the grids and conditional expectation level.
- A second one is achieved at a time step resolution level (file *HighLevel*.py)

In order to test python mpi  :

```
mpirun -np x python test*HighLevelMpi.py
```

with "x" number of mpi processus generated.

# special case for Mac User #
The boost installation can be achieved with a package manager : homebrew for
example will install the missing dependencies (openmpi ...)
In case of a boost compîlation from source, check that "boost mpi" is compiled
and installed. If not : In the directory where the boost file  is decompressed,

```
./boostrap.sh
```

Edit a file user-config.jam and add (don't forget the space between mpi and ;)

```
using mpi ;
```

```
./bjam --user-config=user-config.jam install
```

Most of package can be installed with homebrew (eigen, pybind11).
Only  when activating the SDDP option, Coin utils, osi and clp have to be compiled and installed as the homebrew project only provide binaries.

##################################
## INSTALLATION, TESTS WINDOWS :##
##################################
##################################
# Minimal Requirement

- Win 10
- Visual Studio 2019 

## WINDOWS :  Download CMake for Windows, git for windows, vcpkg and install them          ##


## Compile your own StOpt libs ( when using python bindings)  to use the python library
#####################################################################################

Add python dependance in file vcpkg.json in StOpt directory  if a special python version is needed  

in vcpkg directory,

```
git rev-parse HEAD
```

get back builtin-base string

replace the builtin-base in the vcpkg.json file and modify it  according to your python version



## StOpt compilation and local vcpkg package  installation       ##

In the directory  where StOpt is cloned :

```
mkdir BUILD
cd BUILD
SET VCPKG_FEATURE_FLAGS=versions
cmake -DCMAKE_TOOLCHAIN_FILE=VcpkgDir/vcpkg/scripts/buildsystems/vcpkg.cmake -DBUILD_SDDP=ON -DBUILD_DPCUTS=ON  ../StOpt
```

## installation modification after failure

VCPKG installs a python version but numpy is not available. Update numpy in the python installed after getting "get-pip.py" from

https://pip.pypa.io/en/stable/installation/

and place it in  ./vcpkg_installed/x86-windows/tools/python3

```
cd  vcpkg_installed/x86-windows/tools/python3
python.exe get-pip.py
python.exe  -m pip install numpy
cd ../../../../..

```

And again

```
cmake -DCMAKE_TOOLCHAIN_FILE=VcpkgDir/vcpkg/scripts/buildsystems/vcpkg.cmake -DBUILD_SDDP=ON -DBUILD_DPCUTS=ON  ../StOpt
cmake --build . --config=Release
```


In order to use DPC++ on windows, ninja (https://ninja-build.org/)  should be installed and use the tool chain:

```
cmake -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icx -GNinja -DCMAKE_TOOLCHAIN_FILE=VcpkgDir/vcpkg/scripts/buildsystems/vcpkg.cmake -DBUILD_SDDP=ON -DBUILD_DPCUTS=ON  ../StOpt
cmake --build . --config=Release
```

# alternative : Create your own StOpt c++ vcpkg package (latest python)
#########################################################################


In vcpkg directory   VcpkgDir (absolute path):

```
vcpkg install eigen3 --triplet x64-windows-release
vcpkg install mpi --triplet x64-windows-release
vcpkg install coinutils  --triplet x64-windows-release
vcpkg install coin-or-osi --triplet x64-windows-release
vcpkg install coin-or-clp --triplet x64-windows-release
vcpkg install boost --triplet x64-windows-release
vcpkg install boost-mpi --triplet x64-windows-release
vcpkg install pthreads --triplet x64-windows-release
vcpkg install python3 --triplet x64-windows-release
```

VCPKG installs a python version but numpy is not available. Update numpy in the python installed after getting "get-pip.py" from

https://pip.pypa.io/en/stable/installation/

and place it in  VcpkgDir/installed/x86-windows-release/tools/python3

```
cd  VcpkgDir/installed/x86-windows-release/tools/python3
python.exe get-pip.py
python.exe  -m pip install numpy
cd ../../../../..

```

To install StOpt library from vcpkg registry, clone the git repo  in a directory  :

https://gitlab.com/stochastic-control/stopt-vcpkg-registry-with-python

In vcpkg directory :

```
vcpkg install stopt --overlay-ports=PathToVCPKGRegistry\stopt-vcpkg-registry-with-python\ports\stopt  --triplet x64-windows-release

```

###############################################################################
## Helper for cmake compilation  for projects using StOpt                    ##
###############################################################################

in utils directory, a file FindStOpt.cmake  and a file FindGeners.cmake are provided
to help compilation in projects using cmake


################################################################################
## R package                                                                 ##
###############################################################################

A readme.txt is available in directory StOpt/R


###############################################################################
## Docker
###############################################################################

Thanks to Benjamin Texier a Dockerfile is available  based on ubuntu and anaconda
to test the python interface in a jupyter notebook