1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208
|
// Package storage define abstract storage operation
// Deprecated - please use https://github.com/viant/afs API instead
// This package is frozen and no new functionality will be added, and future removal takes place.
package storage
import (
"fmt"
"io"
"io/ioutil"
"net/url"
"os"
"strings"
)
var DefaultFileMode os.FileMode = 0755
//Service represents abstract way to accessing local or remote storage
type Service interface {
//List returns a list of object for supplied url
List(URL string) ([]Object, error)
//Exists returns true if resource exists
Exists(URL string) (bool, error)
//Object returns a Object for supplied url
StorageObject(URL string) (Object, error)
//Download returns reader for downloaded storage object
Download(object Object) (io.ReadCloser, error)
//DownloadWithURL returns reader for downloaded URL object
DownloadWithURL(URL string) (io.ReadCloser, error)
//Upload uploads provided reader content for supplied storage object.
Upload(URL string, reader io.Reader) error
//Upload uploads provided reader content for supplied storage object.
UploadWithMode(URL string, mode os.FileMode, reader io.Reader) error
//Delete removes passed in storage object
Delete(object Object) error
//Register register schema with provided service
Register(schema string, service Service) error
//Closes storage service
Close() error
}
type storageService struct {
registry map[string]Service
}
func (s *storageService) getServiceForSchema(URL string) (Service, error) {
parsedUrl, err := url.Parse(URL)
if err != nil {
return nil, err
}
if result, found := s.registry[parsedUrl.Scheme]; found {
return result, nil
}
return nil, fmt.Errorf("failed to lookup url schema %v in %v", parsedUrl.Scheme, URL)
}
//List lists all object for passed in URL
func (s *storageService) List(URL string) ([]Object, error) {
service, err := s.getServiceForSchema(URL)
if err != nil {
return nil, err
}
return service.List(URL)
}
//Exists returns true if resource exists
func (s *storageService) Exists(URL string) (bool, error) {
service, err := s.getServiceForSchema(URL)
if err != nil {
return false, err
}
return service.Exists(URL)
}
//StorageObject returns storage object for provided URL
func (s *storageService) StorageObject(URL string) (Object, error) {
service, err := s.getServiceForSchema(URL)
if err != nil {
return nil, err
}
return service.StorageObject(URL)
}
//Download downloads content for passed in object
func (s *storageService) Download(object Object) (io.ReadCloser, error) {
service, err := s.getServiceForSchema(object.URL())
if err != nil {
return nil, err
}
return service.Download(object)
}
//DownloadWithURL downloads content for passed in object URL
func (s *storageService) DownloadWithURL(URL string) (io.ReadCloser, error) {
object, err := s.StorageObject(URL)
if err != nil {
return nil, err
}
return s.Download(object)
}
//Uploads content for passed in URL
func (s *storageService) Upload(URL string, reader io.Reader) error {
service, err := s.getServiceForSchema(URL)
if err != nil {
return err
}
return service.UploadWithMode(URL, 0644, reader)
}
//Uploads content for passed in URL
func (s *storageService) UploadWithMode(URL string, mode os.FileMode, reader io.Reader) error {
service, err := s.getServiceForSchema(URL)
if err != nil {
return err
}
return service.UploadWithMode(URL, mode, reader)
}
//Delete remove storage object
func (s *storageService) Delete(object Object) error {
service, err := s.getServiceForSchema(object.URL())
if err != nil {
return err
}
return service.Delete(object)
}
//Close closes resources
func (s *storageService) Close() error {
for _, service := range s.registry {
err := service.Close()
if err != nil {
return err
}
}
return nil
}
//Register register storage schema
func (s *storageService) Register(schema string, service Service) error {
s.registry[schema] = service
return nil
}
//NewService creates a new storage service
func NewService() Service {
var result = &storageService{
registry: make(map[string]Service),
}
_ = result.Register("file", &fileStorageService{})
_ = result.Register("mem", NewMemoryService())
return result
}
//NewServiceForURL creates a new storage service for provided URL scheme and optional credential file
func NewServiceForURL(URL, credentials string) (Service, error) {
parsedURL, err := url.Parse(URL)
if err != nil {
return nil, err
}
service := NewService()
provider := Registry().Get(parsedURL.Scheme)
if provider != nil {
if len(credentials) > 0 {
credentials = strings.Replace(credentials, "${env.HOME}", os.Getenv("HOME"), 1)
if strings.HasPrefix(credentials, "~") {
credentials = strings.Replace(credentials, "~", os.Getenv("HOME"), 1)
}
}
serviceForScheme, err := provider(credentials)
if err != nil {
return nil, fmt.Errorf("failed lookup service for %v: %v", parsedURL.Scheme, err)
}
err = service.Register(parsedURL.Scheme, serviceForScheme)
if err != nil {
return nil, err
}
} else if parsedURL.Scheme != "file" {
return nil, fmt.Errorf("unsupported scheme %v", URL)
}
return service, nil
}
//Download returns a download reader for supplied URL
func Download(service Service, URL string) (io.ReadCloser, error) {
object, err := service.StorageObject(URL)
if err != nil {
return nil, err
}
return service.Download(object)
}
//DownloadText returns a text for supplied URL
func DownloadText(service Service, URL string) (string, error) {
reader, err := Download(service, URL)
if err != nil {
return "", err
}
defer reader.Close()
content, err := ioutil.ReadAll(reader)
if err != nil {
return "", err
}
return string(content), nil
}
|