File: read_s3.py

package info (click to toggle)
smart-open 7.5.0-2
  • links: PTS, VCS
  • area: main
  • in suites: forky, sid
  • size: 980 kB
  • sloc: python: 8,054; sh: 90; makefile: 14
file content (18 lines) | stat: -rw-r--r-- 449 bytes parent folder | download | duplicates (3) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
import sys

import boto3
import smart_open

urls = [line.strip() for line in sys.stdin]

tp = {}
if 'create_session_and_resource' in sys.argv:
    tp['session'] = boto3.Session()
    tp['resource'] = tp['session'].resource('s3')
elif 'create_resource' in sys.argv:
    tp['resource'] = boto3.resource('s3')
elif 'create_session' in sys.argv:
    tp['session'] = boto3.Session()

for url in urls:
    smart_open.open(url, transport_params=tp).read()